Innovative Integrated Development Solutions

Real-time DevOps Dashboards

Directional Consulting has been a leader in providing monitoring and security dashboards for Enterprise organizations to maintain the visibility and reliability of server applications and infrastructure. In recent years many of our deployments have involved Elasticsearch, Kibana, APM, and their Observability Platform with the enriched Security and AWS plugins.

Live Streaming Log Events and Alerts

Constant monitoring through Filebeat and Elastic AWS Agents running as services on the cloud instances and on-site machines deliver real-time live log streams to the centralized Kibana Dashboards. In addition, text matching patterns and saved KQL (Kibana Query Language) searches instantly identify warning and error conditions by hostname, IP address, and service.

Application Health Checks

Detailed metrics for system services and client applications are automatically collected in real-time to present the overall health status. Knowing how an application or service performs under a load condition is critical to properly allocating resources and defining triggers for scaling.

Detailed Host Overview Metrics

Directional Consulting has extensive experience configuring detailed host monitoring and metrics requirements. Typically the more advanced tracking requires enabling additional libraries and packages, modifying configuration files, and doing some load testing to ensure the application and infrastructure can scale as traffic and demand increase.

Manufacturing Data Acquisition Systems

Directional Consulting has a strong history of designing, developing, and deploying manufacturing data acquisition applications. Our strengths include the development of rich pattern detection and normalizing various file patterns and data constructs, which permits multiple file sets to combine as one data set for any assembly line product.

Manufacturing Traceability

Many manufacturing companies sometimes have extended warranty periods of 10 or 25 years. Our proven techniques for reassembling unique data structures and distributed file sets for each assembly production stage provide the necessary traceability. In addition, the industry-standard OLAP (Online Analytical Processing) database design provides a time, location, and value for every single data cell and correlates it into master tables. Any organizational unit can analyze the master tables by selecting the columns of interest and creating detailed reports.

Device Control Systems

In today's IoT-connected devices and applications, there are many challenges to real-time performance and eliminating latency. Directional Consulting has solid experience working with low-level protocols and server communication gateways using MQTT and Web Socket protocols. Our specialty is designing handshakes and pattern detection to leverage the application's specific needs and the connected hardware.

Real-time IoT Device Information

Shown to the right is a high-level overview of a manufacturing execution system with automatic data acquisition. The real-time component is built with Node.JS, MongoDB, MQTT, and Web Sockets to facilitate bi-directional communication with all the devices attached. In addition, there is a connection from the frontend MES software to the backend ETL engine running PHP and MySQL to provide correlated master reports.

GPS Application Development

Directional Consulting has considerable experience working with GPS data, tracking devices, and GIS maps. Our expertise includes developing the RTVIP (Real-time Vehicle Information Proxy) for the Santa Clara VTA and their fleets of buses and light rail vehicles. The diagram to the left shows VTA vehicle routes and their early, on-time, or late arrival percentages.

Scalable Application Design

We leveraged a nice balance of performance and cost savings using today's standards like AWS auto-scaling groups. For example, the proxy group will auto-spin additional instances once latency and throughput thresholds are triggered. Then during off-peak hours and late at night when few vehicles are in transit, the group runs only one server.

Modern Design Pattern Principles

Incorporating traditional MVC frameworks with more modern Service and Event-driven architecture has proven to be very powerful in delivering robust applications. In addition, diversity and unity are vital in designing a solid integrated framework for real-time data interchange, display, and communications.

Integrated Services and Events

Directional Consulting is skilled at building data pipelines from various sources of data streams, including XML, JSON, CSV, MQTT, and HTTP response codes. Incorporating service event listeners in application design provides additional business logic layers to be implemented, modified, and easily updated without significant design changes.

Low Latency Application Design

The Transit mobile phone app used by thousands of San Fransisco Bay Area commuters daily relies upon Directional Consulting's design and deployment of a parallel background processing GPS proxy engine for the Santa Clara VTA. As a result, incoming GPS requests are immediately bounced within milliseconds to partner transit agencies like Transit, Google, and the MTA.

Used by thousands of Transit app commuters

The video to the left demonstrates the real-time video capture from my mobile phone while using the Transit app and the less than 2-second delay from the inbound light rail vehicle approaching the platform. Notice how the vehicle doors and the PID (Public Information Display) above the tracks synchronize to the precise moment the train arrives.

Trip Planning and Route Directions

Directional Consulting assisted Santa Clara VTA with upgrading its Open Trip Planner application for multi-modal transportation, meaning walking, bicycling, bus, and light rail routes, including bike trails and pedestrian bridges. The project centered around adding new vector overlays using Node.js and D3 graphing libraries, plus adding additional map options like Google and ESRI maps with terrain, roads, and satellite imagery.

Enhanced Printing and Email Directions

To address the needs of VTA customers without smartphones, Directional Consulting created additional Node.js and puppeteer routines to display and overlay the route on a map with a table of the turn-by-turn directions. Technically this feature took much work to achieve due to the nature of Google Maps. However, the knowledge gained is just one more feature option we offer clients.

Real-time Application Metrics

Our design philosophy includes incorporating an APM Agent directly within the application code from the beginning of design conception. The key benefits are knowing in detail what is happening inside the codebase without the need to inspect debug or system logs. Application performance monitoring means detailed information about processes, invalid data values, warnings, errors, and exceptions.

Key Performance Indicators

Custom application metrics become the key performance indicators by reporting how many transactions per minute, the number of records, connections, successful and failure conditions, application health, and status checks. These data metrics result in incremental improvements in the overall performance and connection times to dependent infrastructure resources.

Dashboard Visualizations

We offer extensive experience in creating custom dashboards to monitor the health and performance of the application in an easy-to-understand visual format. For example, see the screenshot of the RTVIP application with regular traffic. If there are any error conditions like latency or an improper HTTP response, the rows will turn yellow and red, reporting the issue.

Performance Tuning and Optimizations

We incorporate application design principles using actual application and server metrics that identify system bottlenecks, throughput latency issues, and extreme memory conditions. We can improve performance by following a proven process of making small server and application changes based on the metrics and measuring the improvements.

Advanced Database Development Tools

Directional Consulting develops and maintains advanced database tools for MySQL server administration, performance optimization, and automated schema documentation. Shown to the left is an ERD (Entity Relationship Diagram) depicting the foreign keys and indexes of the schema.

Directional Consulting maintains and hosts the MySQL Workbench Data Dictionary plugin publicly on GitHub for anyone to quickly generate automatic documentation from the meta information from their MySQL internal tables.

Relational Database Schema Expertise

Relational database designs provide traceability, deep insights, and correlation of data values across vectors and time series. Server-side automated system jobs collate data records frequently into master tables and reports. In addition, foreign and combination column keys significantly optimize performance, storage, and access times.

Advanced MySQL Performance

Directional Consulting has over 15 years of experience tuning, optimizing, configuring, and deploying MySQL servers. Our past projects include deploying multiple MySQL servers globally at various manufacturing facilities and collecting the data centrally at headquarters. In addition, we monitor and continue tuning the server for higher performance for every deployment using benchmark tools and analyzing primary keys, slow queries, and index utilization.

ODBC for MATLAB, Minitab, Power BI, and Tableau

ODBC (Open Database Connectors) are available for Windows, Mac OS, and Linux, enabling a direct connection to the MySQL OLAP (Online Analytical Processing) engine. The ODBC drivers remove the need for CSV files and are significantly faster due to the direct link to the data records. In addition, using BI (Business Intelligence) tools like MATLAB, Minitab, Power BI, and Tableau enable data scientists and research engineers to have deep insights, perform complex analysis, and generate advanced storytelling graphs.

Functional and Unit Testing

Our design and development processes are TDD - Test Driven Development to ensure application accuracy and error handling conditions. We design the application parallel with authoring test case scenarios for positive, negative, out-of-bounds, and anomaly results. As a result, we provide an extensive code-coverage report with graphs to demonstrate the effectiveness of the test suites.

Continuous Delivery

Directional Consulting follows Semantic Versioning standards, with tagged builds and release milestones using various cloud tools, including GitHub, OpenProject, Docker, and Travis CI. We create Docker containers to simulate all requirements of the application and build process, which is then validated automatically by Travis CI build pipelines. These standards enable us to roll out new releases confidently and roll back if needed.

Agile Software Development Life Cycles

Agile software development methodologies are our specialty, from working in Scrum environments like OpenProject and Jira to managing GitHub Issues, Feature Branches, and swim lanes for project management. We have learned from various projects that the P-R-D (Planning, Research, and Design) phases take approximately 1/3rd of the total project time. The later stages of the SDLC include D-I-M (Development, Integration, and Maintenance), measured in incremental builds and milestones as ready/ready and done/done, promoting solid forward momentum in all software releases.

UAT - User Acceptance Testing

Our process of UAT is straightforward by engaging our stakeholders directly from the concept and planning phases until both manual matrix completions of feature sets and automated unit plus functional test suites complete the final acceptance testing. We aim to be there step-by-step to facilitate the project requirements and verify their functionality as described and built to specifications from the design phases.