As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Network Debugging Powerhouse

Modern browsers come equipped with developer tools that rival standalone IDE debuggers in capability and convenience. Both Chrome and Firefox have robust network monitoring features that allow developers to observe/analyze requests and responses without leaving the browser.

On the basic level, which you’re probably familiar with, these tools include:

• Network monitors: View all HTTP and HTTPS requests, including their headers, payloads, and responses.

• Throttling controls: Simulate slower connections to test performance and debug race conditions.

• Request replay functionality: Modify and resend requests directly from the browser.

While this post focuses on debugging techniques, it's worth noting that these tools are invaluable for performance optimization as well, though that topic warrants its own discussion.

Re-Issuing and Modifying Requests

One of the most powerful debugging features is the ability to re-issue requests. Instead of switching to external tools like cURL or Postman, browsers allow us to modify and resend requests directly.

This lets us quickly test variations of a failing API call to pinpoint issues without leaving the debugging environment. It’s especially useful when we have hard to reproduce issues or deep UI hierarchies.

In Firefox we can right-click any network entry in the Firefox Developer Tools and select "Resend." This opens an editable window where we can change request parameters, such as headers, payloads, or query strings, and resend the request.

Chrome provides similar functionality, though its interface for modifying and resending requests is slightly less direct than Firefox's.

cURL and Postman

Both browsers let you copy a request as a cURL command via the context menu. This is useful for reproducing issues in the terminal or sharing with back-end developers. I use this frequently as part of creating a reproducible issue.

If you prefer Postman, you can copy request headers and payloads from the browser and paste them into Postman to replicate requests.

Throttling and Debugging Race Conditions

Network throttling is a highly underrated feature that can be a game-changer for debugging specific classes of bugs. Both Chrome and Firefox allow developers to simulate various network speeds, from 2G connections to fast 4G.

Why Throttling Matters:

Some bugs only surface when requests arrive out of their expected order. Slowing down the network can help replicate and analyze these situations. Typical examples would be race conditions and related issues.

This is also very useful for simulating real-world conditions. Many users may not have fast or reliable internet connections. Throttling helps you understand how your application behaves in these scenarios.

I use this frequently when testing loading indicators which disappear too quickly when running locally. Instead of adding sleep code into the JavaScript or server code I can simulate slow-loading assets to verify that loading spinners or placeholders appear correctly.

How to Use

In Chrome we Open Developer Tools → Network tab.

We then use the "No throttling" dropdown to select pre-configured speeds or create a custom profile.

In Firefox we have similar functionality is available under the Network Monitor.

Managing State with Storage Tools

Local storage, session storage, and indexedDB often hold data critical to reproducing bugs, especially those tied to specific user states or devices.

Challenges of State Management

Even in incognito mode, state can persist if multiple private windows are open simultaneously. Persistence across sessions is a big challenge in these situations.

Understanding the exact state of a user's local storage can provide insight into seemingly random bugs. Debugging user-specific issues is problematic without control over storage.

In Firefox the dedicated Storage tab in Developer Tools makes it easy to inspect, edit, and delete data from local storage, session storage, cookies, and indexedDB.

In Chrome the Application tab consolidates all storage options, including the ability to clear specific caches or edit entries manually.

This functionality has many powerful uses for debugging:

1. Inject Debug Information: Tools like these let us manually add or modify storage data to simulate edge cases or specific user conditions.

2. Share Local State: Users can export their local storage, cookies, or indexedDB entries, allowing developers to reproduce issues locally.

3. Clear Cache Strategically: Clear only the relevant entries instead of a blanket cache clear, preserving useful state for debugging.

Analyzing Request and Response Headers

Request and response headers often hold the key to understanding network issues. We can use the network monitor to inspect:

• Authorization headers: Check for missing or malformed tokens.

• CORS headers: Verify that the server allows requests from your domain. These are some of the most painful type of http bugs. Reviewing these headers can be a lifesaver. If requests fail with CORS errors, inspect the Access-Control-Allow-Origin header in the response.

• Cache-Control headers: Ensure proper caching behavior for your resources.

These tools are especially useful when debugging missing headers: Look for required headers like Content-Type or Authorization. Debugging Authentication: Use the "Copy as cURL" feature to test API calls with modified headers directly in the terminal.

Debugging in Incognito Mode: Limitations and Best Practices

Incognito mode can help isolate issues by providing a clean slate, however it has some limitations. Multiple incognito windows share the same state, which can lead to unintentional persistence of local data.

I suggest using storage management tools to manually clear or modify local data instead of relying solely on incognito mode. Keep only one incognito window open during testing to avoid unintended state sharing.

Connecting the Front-End to the Database

The front-end is often a transition point between user interaction and back-end data processing. While this post focuses on debugging the network layer, it's important to remember that:

• Network issues often manifest due to back-end problems (e.g., a database error resulting in a 500 Internal Server Error).

• Front-end developers should collaborate closely with back-end engineers to trace issues across the stack.

We can use custom response headers to include diagnostic information from the back end, such as query execution time or error codes. We can leverage server logs in conjunction with front-end debugging to get a complete picture of the issue.

Final Word

Browser developer tools are indispensable for debugging network communication issues, offering features like request replay, throttling, and storage management that simplify the debugging process. By mastering these tools, front-end developers can efficiently identify and resolve issues, ensuring a smoother user experience.

With the techniques and tips outlined in this post, you'll be better equipped to tackle network debugging challenges head-on. As you grow more familiar with these tools, you'll find them invaluable not only for debugging but also for improving your development workflow.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Understanding Front-End Logging vs. Back-End Logging

Front-end logging differs significantly from back-end logging, and understanding this distinction is crucial. Unlike back-end systems, where persistent logs are vital for monitoring and debugging, the fluid nature of front-end development introduces different challenges. When debugging backends I’d often go for tracepoints which are far superior in that setting. However the front-end with its constant need to refresh, reload, contexts switch etc. is a very different beast. In the front-end relying heavily on elaborate logging mechanisms can become cumbersome.

While tracepoints remain superior to basic print statements, the continuous testing and browser reloading in front-end workflows lessen their advantage. Moreover, features like logging to a file or structured ingestion are rarely useful in the browser, diminishing the need for a comprehensive logging framework. However, using a logger is still considered best practice over the typical Console.log for long term logging… For short term logging Console.log has some tricks up its sleeve.

Leveraging Console Log Levels

One of the hidden gems of the browser console is its support for log levels, which is a significant step up from rudimentary print statements. The console provides five levels:

• log: Standard logging

• debug: Same as log but used for debugging purposes

• info: Informative messages, often rendered like log/debug

• warn: Warnings that might need attention

• error: Errors that have occurred

While log and debug can be indistinguishable, these levels allow for a more organized and filtered debugging experience. Browsers enable filtering the output based on these levels, mirroring the capabilities of server-side logging systems and allowing you to focus on relevant messages.

Customizing Console Output with CSS

Front-end development allows for creative solutions, and logging is no exception. Using CSS styles in the console can make logs more visually distinct. By utilizing %c in a console message, you can apply custom CSS:

console.customLog = function(msg) {
    console.log("%c" + msg,"color:black;background:pink;font-family:system-ui;font-size:4rem;-webkit-text-stroke: 1px black;font-weight:bold")
}
console.customLog("Dazzle")

This approach is helpful when you need to make specific logs stand out or organize output visually. You can use multiple %c substitutions to apply various styles to different parts of a log message.

Stack Tracing with console.trace()

The console.trace() method can print a stack trace at a particular location, which can sometimes be helpful for understanding the flow of your code. However, due to JavaScript’s asynchronous behavior, stack traces aren’t always as straightforward as in back-end debugging. Still, in specific scenarios, such as synchronous code segments or event handling, it can be quite valuable.

Assertions for Design-by-Contract

Assertions in front-end code allow developers to enforce expectations and promote a “fail-fast” mentality. Using Console.assert(), you can test conditions:

console.assert(x > 0, 'x must be greater than zero');

In the browser, a failed assertion appears as an error, similar to console.error. An added benefit is that assertions can be stripped from production builds, removing any performance impact. This makes assertions a great tool for enforcing design contracts during development without compromising production efficiency.

Printing Tables for Clearer Data Visualization

When working with arrays or objects, displaying data as tables can significantly enhance readability. The console.table() method allows you to output structured data easily:

console.table(["Simple Array", "With a few elements", "in line"])

This method is especially handy when debugging arrays of objects, presenting a clear, tabular view of the data and making complex data structures much easier to understand.

Copying Objects to the Clipboard

Debugging often involves inspecting objects, and the copy(object) method allows you to copy an object’s content to the clipboard for external use. This feature is useful when you need to transfer data or analyze it outside the browser.

Inspecting with console.dir() and dirxml()

The console.dir() method provides a more detailed view of objects, showing their properties as you’d see in a debugger. This is particularly helpful for inspecting DOM elements or exploring API responses. Meanwhile, console.dirxml() allows you to view objects as XML, which can be useful when debugging HTML structures.

Counting Function Calls

Keeping track of how often a function is called or a code block is executed can be crucial. The console.count() method tracks the number of times it’s invoked, helping you verify that functions are called as expected:

function myFunction() {
    console.count('myFunction called');
}

You can reset the counter using console.countReset(). This simple tool can help you catch performance issues or confirm the correct execution flow.

Organizing Logs with Groups

To prevent log clutter, use console groups to organize related messages. console.group() starts a collapsible log section, and console.groupEnd() closes it:

console.group('My Group');
console.log('Message 1');
console.log('Message 2');
console.groupEnd();

Grouping makes it easier to navigate complex logs and keeps your console clean.

Chrome-Specific Debugging Features

Monitoring Functions: Chrome’s monitor() method logs every call to a function, showing the arguments and enabling a method-tracing experience.

Monitoring Events: Using monitorEvents(), you can log events on an element. This is useful for debugging UI interactions. For example, monitorEvents(window, 'mouseout') logs only mouseout events.

Querying Object Instances: queryObjects(Constructor) lists all objects created with a specific constructor, giving you insights into memory usage and object instantiation.

Final Word

Front-end debugging tools have come a long way. These tools provide a rich set of features that go far beyond simple console.log() statements. From log levels and CSS styling to assertions and event monitoring, mastering these techniques can transform your debugging workflow.

If you read this post as part of my series you will notice a big change in my attitude toward debugging when we reached the front-end. Front-end debugging is very different when compared to backend debugging. When debugging the backend I’m vehemently against code changes for debugging (e.g. println debugging), but on the front-end this can be a reasonable hack. The change in environment justifies it. The short lifecycle, the single user use case and the risk is smaller.

While there are many transferrable skills we pick up while debugging, it’s important to remain flexible in our attitude. Next time we will discuss networking and storage debugging on the front-end.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Instant Debugging with the debugger Keyword

A cool yet powerful tool in JavaScript is the debugger keyword. Instead of simply printing a stack trace, we can use this keyword to launch the debugger directly at the line of interest. That is a fantastic tool that instantly brings your attention to a bug, I often use it in my debug builds of the front-end instead of just printing an error log.

How to Use It: Place the debugger keyword within your code, particularly within error-handling methods. When the code execution hits this line, it automatically pauses, allowing you to inspect the current state, step through code, and understand what's going wrong.

Notice that while this is incredibly useful during development, we must remember to remove or conditionally exclude debugger statements in production environments. A release build should not include these calls in a production site live environment.

Triggering Debugging from the Console

Modern browsers allow you to invoke debugging directly from the console, adding an additional layer of flexibility to your debugging process.

Example: By using the debug(functionName) command in the console, you can set a breakpoint at the start of the specified function. When this function is subsequently invoked, the execution halts, sending you directly into the debugger.

function hello(name) {
    Console.log("Hello " + name)
}
debug(hello)
hello("Shai")

This is particularly useful when you want to start debugging without modifying the source code, or when you need to inspect a function that’s only defined in the global scope.

DOM Breakpoints: Monitoring DOM Changes

DOM breakpoints are an advanced feature in Chrome and Firebug (Firefox plugin) that allow you to pause execution when a specific part of the DOM is altered.

To use it we can right-click on the desired DOM element, select “Break On,” and choose the specific mutation type you are interested in (e.g., subtree modifications, attribute changes, etc.).

DOM breakpoints are extremely powerful for tracking down issues where DOM manipulation causes unexpected results, such as dynamic content loading or changes in the user interface that disrupt the intended layout or functionality. Think of them like field breakpoints we discussed in the past.

These breakpoints complement traditional line and conditional breakpoints, providing a more granular approach to debugging complex front-end issues. This is a great tool to use when the DOM is manipulated by an external dependency.

XHR Breakpoints: Uncovering Hidden Network Calls

Understanding who initiates specific network requests can be challenging, especially in large applications with multiple sources contributing to a request. XHR (XMLHttpRequest) breakpoints provide a solution to this problem.

In Chrome or Firebug, set an XHR breakpoint by specifying a substring of the URI you wish to monitor. When a request matching this pattern is made, the execution stops, allowing you to investigate the source of the request.

This tool is invaluable when dealing with dynamically generated URIs or complex flows where tracking the origin of a request is not straightforward.

Notice that you should be selective with the filters you set; leaving the filter blank will cause the breakpoint to trigger on all XHR requests, which can become overwhelming.

Simulating Environments for Debugging

Sometimes, the issues you need to debug are specific to certain environments, such as mobile devices or different geographical locations. Chrome and Firefox offer several simulation tools to help you replicate these conditions on your desktop.

• Simulating User Agents: Change the browser’s user agent to mimic different devices or operating systems. This can help you identify platform-specific issues or debug server-side content delivery that varies by user agent.

• Geolocation Spoofing: Modify the browser’s reported location to test locale-specific features or issues. This is particularly useful for applications that deliver region-specific content or services.

• Touch and Device Orientation Emulation: Simulate touch events or change the device orientation to see how your application responds to mobile-specific interactions. This is crucial for ensuring a seamless user experience across all devices.

These are things that are normally very difficult to reproduce. E.g. touch related issues are often challenging to debug on the device. By simulating them on the desktop browser we can shorten the debug cycle and use the tooling available on the desktop.

Debugging Layout and Style Issues

CSS and HTML bugs can be particularly tricky, often requiring a detailed examination of how elements are rendered and styled.

Inspect Element: The "inspect element" tool is the cornerstone of front-end debugging, allowing you to view and manipulate the DOM and CSS in real-time. As you make changes, the page updates instantly, providing immediate feedback on your tweaks.

Addressing Specificity Issues: One common problem is CSS specificity, where a more specific selector overrides the styles you intend to apply. The inspect element view highlights overridden styles, helping you identify and resolve conflicts.

Firefox vs. Chrome: While both browsers offer robust tools, they have different approaches to organizing these features. Firefox’s interface may seem more straightforward, with fewer tabs, while Chrome organizes similar tools under various tabs, which can either streamline your workflow or add complexity, depending on your preference.

Final Word

There are many front-end tools that I want to discuss in the coming posts. I hope you picked up a couple of new debugging tricks in this first part.

Front-end debugging requires deep understanding of browser tools and JavaScript capabilities. By mastering the techniques outlined in this post—instant debugging with the debugger keyword, DOM and XHR breakpoints, environment simulation, and layout inspection—you can significantly enhance your debugging efficiency and deliver more robust, error-free web applications.

Notice that this is an introductory post focusing mostly on the front end debugging aspects, in the following posts I will dig deeper into the less familiar capabilities in front end debugging.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Full Stack Development, A Shifting Definition

The definition of full stack development is as fluid as the technology stacks themselves. Traditionally, full stack developers were defined as those who could work on both the frontend and backend of an application. However, as the industry evolves, this definition has expanded to include aspects of operations (OPS) and configuration. The modern full stack developer is expected to submit pull requests that cover all parts required to implement a feature—backend, database, frontend, and configuration. While this does not make them an expert in all these areas, it does require them to navigate across domains, often relying on domain experts for guidance.

I've heard it said that full stack developers are:

Jack of all trades, master of none.

However, the full quote probably better represents the reality:

Jack of all trades, master of none, but better than a master of one.

The Full Stack Debugging Approach

Just as full stack development involves working across various domains, full stack debugging requires a similar approach. A symptom of a bug may manifest in the frontend, but its root cause could lie deep within the backend or database layers. Full stack debugging is about tracing these issues through the layers and isolating them as quickly as possible. This is no easy task, especially when dealing with complex systems where multiple layers interact in unexpected ways. The key to successful full stack debugging lies in understanding how to track an issue through each layer of the stack and identifying common pitfalls that developers may encounter.

Frontend Debugging: Tools and Techniques

It isn't "Just Console.log"

Frontend developers are often stereotyped as relying solely on Console.log for debugging. While this method is simple and effective for basic debugging tasks, it falls short when dealing with the complex challenges of modern web development. The complexity of frontend code has increased significantly, making advanced debugging tools not just useful, but necessary. Yet, despite the availability of powerful debugging tools, many developers continue to shy away from them, clinging to old habits.

The Power of Developer Tools

Modern web browsers come equipped with robust developer tools that offer a wide range of capabilities for debugging frontend issues. These tools, available in browsers like Chrome and Firefox, allow developers to inspect elements, view and edit HTML and CSS, monitor network activity, and much more. One of the most powerful, yet underutilized, features of these tools is the JavaScript debugger.

The debugger allows developers to set breakpoints, step through code, and inspect the state of variables at different points in the execution. However, the complexity of frontend code, particularly when it has been obfuscated for performance reasons, can make debugging a challenging task.

We can launch the browser tools on Firefox using this menu:

On Chrome we can use this option:

I prefer working with Firefox, I find their developer tools more convenient but both browsers have similar capabilities. Both have fantastic debuggers (as you can see with the Firefox debugger below), unfortunately many developers limit themselves to console printing instead of exploring this powerful tool.

Tackling Code Obfuscation

Code obfuscation is a common practice in frontend development, employed to protect proprietary code and reduce file sizes for better performance. However, obfuscation also makes the code difficult to read and debug. Fortunately, both Chrome and Firefox developer tools provide a feature to de-obfuscate code, making it more readable and easier to debug. By clicking the curly brackets button in the toolbar, developers can transform a single line of obfuscated code into a well-formed, debuggable file.

Another important tool in the fight against obfuscation is the source map. Source maps are files that map obfuscated code back to its original source code, including comments. When generated and properly configured, source maps allow developers to debug the original code instead of the obfuscated version. In Chrome, this feature can be enabled by ensuring that "Enable JavaScript source maps" is checked in the developer tools settings.

You can use code like this in the JavaScript file to point at the sourcemap file:

//@sourceMappingURL=myfile.js.map

For this to work in Chrome we need to ensure that "Enable JavaScript source maps" is checked in the settings. Last I checked it was on by default but it doesn't hurt to verify:

Debugging Across Layers

Isolating Issues Across the Stack

In full stack development, issues often manifest in one layer but originate in another. For example, a frontend error might be caused by a misconfigured backend service or a database query that returns unexpected results. Isolating the root cause of these issues requires a methodical approach, starting from the symptom and working backward through the layers.

A common strategy is to reproduce the issue in a controlled environment, such as a local development setup, where each layer of the stack can be tested individually. This helps to narrow down the potential sources of the problem. Once the issue has been isolated to a specific layer, developers can use the appropriate tools and techniques to diagnose and resolve it.

The Importance of System-Level Debugging

Full stack debugging is not limited to the application code. Often, issues arise from the surrounding environment, such as network configurations, third-party services, or hardware limitations. A classic example of this that we ran into a couple of years ago was a production problem where a WebSocket connection would frequently disconnect. After extensive debugging, Steve discovered that the issue was caused by the CDN provider (CloudFlare) timing out the WebSocket after two minutes—something that could only be identified by debugging the entire system, not just the application code.

System-level debugging requires a broad understanding of how different components of the infrastructure interact with each other. It also involves using tools that can monitor and analyze the behavior of the system as a whole, such as network analyzers, logging frameworks, and performance monitoring tools.

Embracing Complexity

Full stack debugging is inherently complex, as it requires developers to navigate multiple layers of an application, often dealing with unfamiliar technologies and tools. However, this complexity also presents an opportunity for growth. By embracing the challenges of full stack debugging, developers can expand their knowledge and become more versatile in their roles.

One of the key strengths of full stack development is the ability to collaborate with domain experts. When debugging an issue that spans multiple layers, it is important to leverage the expertise of colleagues who specialize in specific areas. This collaborative approach not only helps to resolve issues more efficiently but also fosters a culture of knowledge sharing and continuous learning within the team.

As tools continue to evolve, so too do the tools and techniques available for debugging. Developers should strive to stay up-to-date with the latest advancements in debugging tools and best practices. Whether it’s learning to use new features in browser developer tools or mastering system-level debugging techniques, continuous learning is essential for success in full stack development.

Conclusion

Full stack debugging is a critical skill for modern developers, we mistakenly think it requires deep understanding of both the application and its surrounding environment. I disagree... By mastering the tools and techniques discussed in this post/upcoming posts, developers can more effectively diagnose and resolve issues that span multiple layers of the stack. Whether you’re dealing with obfuscated frontend code, misconfigured backend services, or system-level issues, the key to successful debugging lies in a methodical, collaborative approach.

You don't need to understand every part of the system, just the ability to eliminate the impossible.

Before I proceed I think it's important to disclose a bias: I am personally not a huge fan of Serverless or PaaS after I was burned badly by PaaS in the past. However, some smart people like Adam swear by it so I should keep an open mind.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Introduction to Serverless Computing

Serverless computing, often referred to as Function as a Service (FaaS), allows developers to build and run applications without managing servers. In this model, cloud providers automatically handle the infrastructure, scaling, and management tasks, enabling developers to focus purely on writing and deploying code. Popular serverless platforms include AWS Lambda, Azure Functions, and Google Cloud Functions.

In contrast, Platform as a Service (PaaS) offers a more managed environment where developers can deploy applications but still need to configure and manage some aspects of the infrastructure. PaaS solutions, such as Heroku and Google App Engine, provide a higher level of abstraction than Infrastructure as a Service (IaaS) but still require some server management.

Kubernetes, which we recently discussed, is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. While Kubernetes offers powerful capabilities for managing complex, multi-container applications, it requires significant expertise to set up and maintain. Serverless computing simplifies this by removing the need for container orchestration and management altogether.

The "catch" is two fold:

• Serverless programming removes the need to understand the servers but also removes the ability to rely on them resulting in more complex architectures.

• Pricing starts off cheap. Practically free. It can quickly escalate especially in case of an attack or misconfiguration.

Challenges of Serverless Debugging

While serverless architectures offer some benefits, they also introduce unique debugging challenges. The primary issues stem from the inherent complexity and distributed nature of serverless environments. Here are some of the most pressing challenges.

Disconnected Environments

One of the major hurdles in serverless debugging is the lack of consistency between development, staging, and production environments. While traditional development practices rely on these separate environments to test and validate code changes, serverless architectures often complicate this process. The differences in configuration and scale between these environments can lead to bugs that only appear in production, making them difficult to reproduce and fix.

Lack of Standardization

The serverless ecosystem is highly fragmented, with various vendors offering different tools and frameworks. This lack of standardization can make it challenging to adopt a unified debugging approach. Each platform has its own set of practices and tools, requiring developers to learn and adapt to multiple environments.

This is slowly evolving with some platforms gaining traction, but since this is a vendor driven industry there are many edge cases.

Limited Debugging Tools

Traditional debugging tools, such as step-through debugging and breakpoints, are often unavailable in serverless environments. The managed and controlled nature of serverless functions restricts access to these tools, forcing developers to rely on alternative methods, such as logging and remote debugging.

Concurrency and Scale

Serverless functions are designed to handle high concurrency and scale seamlessly. However, this can introduce issues that are hard to reproduce in a local development environment. Bugs that manifest only under specific concurrency conditions or high load are particularly challenging to debug.

Notice that when I discuss concurrency here I'm often referring to race conditions between separate services.

Effective Strategies for Serverless Debugging

Despite these challenges, several strategies can help make serverless debugging more manageable. By leveraging a combination of local debugging, feature flags, staged rollouts, logging, idempotency, and Infrastructure as Code (IaC), developers can effectively diagnose and fix issues in serverless applications.

Local Debugging with IDE Remote Capabilities

While serverless functions run in the cloud, you can simulate their execution locally using tools like AWS SAM (Serverless Application Model). This involves setting up a local server that mimics the cloud environment, allowing you to run tests and perform basic trial-and-error debugging.

To get started, you need to install Docker or Docker Desktop, create an AWS account, and set up the AWS SAM CLI. Deploy your serverless application locally using the SAM CLI, which enables you to run the application and simulate Lambda functions on your local machine. Configure your IDE for remote debugging, launching the application in debug mode, and connecting your debugger to the local host. Set breakpoints to step through the code and identify issues.

Using Feature Flags for Debugging

Feature flags allow you to enable or disable parts of your application without deploying new code. This can be invaluable for isolating issues in a live environment. By toggling specific features on or off, you can narrow down the problematic areas and observe the application’s behavior under different configurations.

Implementing feature flags involves adding conditional checks in your code that control the execution of specific features based on the flag’s status. Monitoring the application with different flag settings helps identify the source of bugs and allows you to test fixes without affecting the entire user base.

This is essentially "debugging in production". Working on a new feature?

Wrap it in a feature flag which is effectively akin to wrapping the entire feature (client and server) in if statements. You can then enable it conditionally globally or on a per user basis. This means you can test the feature, enable or disable it based on configuration without redeploying the application.

Staged Rollouts and Canary Deployments

Deploying changes incrementally can help catch bugs before they affect all users. Staged rollouts involve gradually rolling out updates to a small percentage of users before a full deployment. This allows you to monitor the performance and error logs of the new version in a controlled manner, catching issues early.

Canary deployments take this a step further by deploying new changes to a small subset of instances (canaries) while the rest of the system runs the stable version. If issues are detected in the canaries, you can roll back the changes without impacting the majority of users. This method limits the impact of potential bugs and provides a safer way to introduce updates. This isn't great as in some cases some demographics might be more reluctant to report errors. However, for server side issues this might make sense as you can see the impact based on server logs and metrics.

Comprehensive Logging

Logging is one of the most common and essential tools for debugging serverless applications. I wrote and spoke a lot about logging in the past. By logging all relevant data points, including inputs and outputs of your functions, you can trace the flow of execution and identify where things go wrong.

However, excessive logging can increase costs, as serverless billing is often based on execution time and resources used. It’s important to strike a balance between sufficient logging and cost efficiency. Implementing log levels and selectively enabling detailed logs only when necessary can help manage costs while providing the information needed for debugging.

I talk about striking the delicate balance between debuggable code, performance and cost with logs in the following video. Notice that this is a general best practice and not specific to serverless.

Embracing Idempotency

Idempotency, a key concept from functional programming, ensures that functions produce the same result given the same inputs, regardless of the number of times they are executed. This simplifies debugging and testing by ensuring consistent and predictable behavior.

Designing your serverless functions to be idempotent involves ensuring that they do not have side effects that could alter the outcome when executed multiple times. For example, including timestamps or unique identifiers in your requests can help maintain consistency. Regularly testing your functions to verify idempotency can make it easier to pinpoint discrepancies and debug issues.

Testing is always important but in serverless and complex deployments it becomes critical. Awareness and embrace of idempotency allows for more testable code and easier to reproduce bugs.

Debugging a Lambda Application Locally with AWS SAM

Debugging serverless applications, particularly AWS Lambda functions, can be challenging due to their distributed nature and the limitations of traditional debugging tools. However, AWS SAM (Serverless Application Model) provides a way to simulate Lambda functions locally, enabling developers to test and debug their applications more effectively. I will use it as a sample to explore the process of setting up a local debugging environment, running a sample application, and configuring remote debugging.

Setting Up the Local Environment

Before diving into the debugging process, it's crucial to set up a local environment that can simulate the AWS Lambda environment. This involves a few key steps:

1. Install Docker: Docker is required to run the local simulation of the Lambda environment. You can download Docker or Docker Desktop from the official Docker website.

2. Create an AWS Account: If you don't already have an AWS account, you need to create one. Follow the instructions on the AWS account creation page.

3. Set Up AWS SAM CLI: The AWS SAM CLI is essential for building and running serverless applications locally. You can install it by following the AWS SAM installation guide.

Running the Hello World Application Locally

To illustrate the debugging process, let's use a simple "Hello World" application. The code for this application can be found in the AWS Hello World tutorial.

1. Deploy Locally: Use the SAM CLI to deploy the Hello World application locally. This can be done with the following command:

    sam local start-api
   

   This command starts a local server that simulates the AWS Lambda cloud environment.

2. Trigger the Endpoint: Once the local server is running, you can trigger the endpoint using a curl command:

    curl http://localhost:3000/hello
   

   This command sends a request to the local server, allowing you to test the function's response.

Configuring Remote Debugging

While running tests locally is a valuable step, it doesn't provide full debugging capabilities. To debug the application, you need to configure remote debugging. This involves several steps.

First we need to start the application in debug mode using the following SAM command:

sam local invoke -d 5858

This command pauses the application and waits for a debugger to connect.

Next we need to configure the IDE for remote debugging. We start by setting up the IDE to connect to the local host for remote debugging. This typically involves creating a new run configuration that matches the remote debugging settings.

We can now set breakpoints in the code where we want the execution to pause. This allows us to step through the code and inspect variables and application state just like in any other local application.

We can test this by invoking the endpoint e.g. using curl. With the debugger connected we would stop on the breakpoint like any other tool:

curl http://localhost:3000/hello

The application will pause at the breakpoints you set, allowing you to step through the code.

Handling Debugger Timeouts

One significant challenge when debugging Lambda functions is the quick timeout setting. Lambda functions are designed to execute quickly, and if they take too long, the costs can become prohibitive. By default, the timeout is set to a short duration, but you can configure this in the template.yaml file e.g.:

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.lambdaHandler
      Timeout: 60  # timeout in seconds

After updating the timeout value, re-issue the sam build command to apply the changes.

In some cases, running the application locally might not be enough. You may need to simulate running on the actual AWS stack to get more accurate debugging information. Solutions like SST (Serverless Stack) or MerLoc can help achieve this, though they are specific to AWS and relatively niche.

Final Word

Serverless debugging requires a combination of strategies to effectively identify and resolve issues. While traditional debugging methods may not always apply, leveraging local debugging, feature flags, staged rollouts, comprehensive logging, idempotency, and IaC can significantly improve your ability to debug serverless applications. As the serverless ecosystem continues to evolve, staying adaptable and continuously updating your debugging techniques will be key to success.

Debugging serverless applications, particularly AWS Lambda functions, can be complex due to their distributed nature and the constraints of traditional debugging tools. However, by leveraging tools like AWS SAM, you can simulate the Lambda environment locally and use remote debugging to step through your code. Adjusting timeout settings and considering advanced simulation tools can further enhance your debugging capabilities.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Identifying Configuration Issues

When you encounter configuration issues in Kubernetes, the first place to check is the status column using the kubectl get pods command. Common errors manifest here, requiring further inspection with kubectl describe pod.

$ kubectl get pods
NAME                     READY    STATUS     RESTARTS   AGE 
my-first-pod-id-xxxx      1/1     Running    0          13s
my-second-pod-id-xxxx     1/1     Running    0          13s

Common Causes and Solutions

Insufficient Resources: Notice that this means resources for the POD itself and not resources within the container. It means the hardware or surrounding VM is hitting a limit.

Symptom: Pods fail to schedule due to resource constraints.

Solution: Scale up the cluster by adding more nodes to accommodate the resource requirements.

Volume Mounting Failures:

Symptom: Pods cannot mount volumes correctly.

Solution: Ensure storage is defined accurately in the pod specification and check the storage class and Persistent Volume (PV) configurations.

Detailed Investigation Steps

We can use kubectl describe pod: This command provides a detailed description of the pod, including events that have occurred. By examining these events, we can pinpoint the exact cause of the issue.

Another important step is resource quota analysis. Sometimes, resource constraints are due to namespace-level resource quotas. Use kubectl get resourcequotas to check if quotas are limiting pod creation.

Dealing with Image Pull Errors

Errors like ErrImagePull or ImagePullBackOff indicate issues with fetching container images. These errors are typically related to image availability or access permissions.

Troubleshooting Steps

The first step is checking the image name which we can do with the following command:

docker pull <image-name>

We then need to verify the image name for typos or invalid characters. I pipe the command through grep to verify the name is 100% identical, some typos are just notoriously hard to spot.

Credentials can also be a major pitfall. E.g. an authorization failure when pulling images from private repositories.

We must ensure that Docker registry credentials are correctly configured in Kubernetes secrets.

Network configuration should also be reviewed. Ensure that the Kubernetes nodes have network access to the Docker registry. Network policies or firewall rules might block access.

There are quite a few additional pitfalls such as problems with image tags. Ensure you are using the correct image tags. Latest tags might not always point to the expected image version.

If you're using a private registry you might be experiencing access issues. Make sure your credentials are up-to-date and the registry is accessible from all nodes in all regions.

Handling Node Issues

Node-related errors often point to physical or virtual machine issues. These issues can disrupt the normal operation of the Kubernetes cluster and need prompt attention.

To check node status use the command:

kubectl get nodes

We can then identify problematic nodes in the resulting output.

It's a cliché but sometimes rebooting nodes is the best solution to some problems. We can reboot the affected machine or VM. Kubernetes should attempt to "self-heal" and recover within a few minutes.

To investigate node conditions we can use the command:

kubectl describe node <node-name>

We should look for conditions such as MemoryPressure, DiskPressure, or NetworkUnavailable. These conditions provide clues about the underlying issue we should address in the node.

Preventive Measures

Node monitoring should be used to with tools such as Prometheus, Grafana to keep an eye on node health and performance. These work great for the low level Kubernetes related issues, we can also use them for high level application issues.

There are some automated healing tools such as the Kubernetes Cluster Autoscaler that we can leverage to automatically manage the number of nodes in your cluster based on workload demands. Personally, I'm not a huge fan as I'm afraid of a cascading failure that would trigger additional resource consumption.

Managing Missing Configuration Keys or Secrets

Missing configuration keys or secrets are common issues that disrupt Kubernetes deployments. Proper management of these elements is crucial for smooth operation.

We need to use ConfigMaps and secrets. These let us store configuration values and sensitive information securely. To avoid that we need to ensure that ConfigMaps and Secrets are correctly referenced in your pod specifications.

Inspect pod descriptions using the command:

kubectl describe pod <pod-name>

Review the output and look for missing configuration details. Rectify any misconfigurations.

ConfigMap and secret creation can be verified using the command:

kubectl get configmaps

and:

kubectl get secrets

Ensure that the required ConfigMaps and Secrets exist in the namespace and contain the expected data.

It's best to keep non-sensitive parts of ConfigMaps in version control while excluding Secrets for security. Furthermore, you should use different ConfigMaps and Secrets for different environments (development, staging, production) to avoid configuration leaks.

Utilizing Buildg for Interactive Debugging

Buildg is a relatively new tool that enhances the debugging process for Docker configurations by allowing interactive debugging.

It provides Interactive Debugging for configuration issues in a way that's similar to a standard debugging. It lets us step through the Dockerfile stages and set breakpoints. Buildg is compatible with VSCode and other IDEs via the Debug Adapter Protocol (DAP).

Buildg lets us inspect container state at each stage of the build process to identify issues early.

To install buildg follow the instructions on the Buildg GitHub page.

Conclusion

Debugging Kubernetes can be challenging, but with the right knowledge and tools, developers can effectively identify and resolve common issues. By understanding configuration problems, image pull errors, node issues, and the importance of ConfigMaps and Secrets, developers can contribute to more robust and reliable Kubernetes deployments. Tools like Buildg offer promising advancements in interactive debugging, further bridging the gap between development and operations.

As Kubernetes continues to evolve, staying informed about new tools and best practices will be essential for successful application management and deployment. By proactively addressing these common issues, developers can ensure smoother, more efficient Kubernetes operations, ultimately leading to more resilient and scalable applications.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

The Immutable Nature of Containers

One of the fundamental principles of Kubernetes is the immutability of container instances. This means that once a container is running, it shouldn't be altered. Modifying containers on the fly can lead to inconsistencies and unpredictable behavior, especially as Kubernetes orchestrates the lifecycle of these containers, replacing them as needed. Imagine trying to diagnose an issue only to realize that the container you’re investigating has been modified, making it difficult to reproduce the problem consistently.

The idea behind this immutability is to ensure that every instance of a container is identical to any other instance. This consistency is crucial for achieving reliable, scalable applications. If you start modifying containers, you undermine this consistency, leading to a situation where one container behaves differently from another, even though they are supposed to be identical.

The Limitations of kubectl exec

We often start our journey in Kubernetes with commands such as:

$ kubectl -- exec -ti <pod-name>

This logs into a container and feels like accessing a traditional server with SSH. However, this approach has significant limitations. Containers often lack basic diagnostic tools—no vim, no traceroute, sometimes not even a shell. This can be a rude awakening for those accustomed to a full-featured Linux environment. Additionally, if a container crashes, kubectl exec becomes useless as there's no running instance to connect to. This tool is insufficient for thorough debugging, especially in production environments.

Consider the frustration of logging into a container only to find out that you can't even open a simple text editor to check configuration files. This lack of basic tools means that you are often left with very few options for diagnosing problems. Moreover, the minimalistic nature of many container images, designed to reduce their attack surface and footprint, exacerbates this issue.

Avoiding Direct Modifications

While it might be tempting to install missing tools on-the-fly using commands like apt-get install vim, this practice violates the principle of container immutability. In production, installing packages dynamically can introduce new dependencies, potentially causing application failures. The risks are high, and it's crucial to maintain the integrity of your deployment manifests, ensuring that all configurations are predefined and reproducible.

Imagine a scenario where a quick fix in production involves installing a missing package. This might solve the immediate problem but could lead to unforeseen consequences. Dependencies introduced by the new package might conflict with existing ones, leading to application instability. Moreover, this approach makes it challenging to reproduce the exact environment, which is vital for debugging and scaling your application.

Enter Ephemeral Containers

The solution to the aforementioned problems lies in ephemeral containers. Kubernetes allows the creation of these temporary containers within the same pod as the application container you need to debug. These ephemeral containers are isolated from the main application, ensuring that any modifications or tools installed do not impact the running application.

Ephemeral containers provide a way to bypass the limitations of kubectl exec without violating the principles of immutability and consistency. By launching a separate container within the same pod, you can inspect and diagnose the application container without altering its state. This approach preserves the integrity of the production environment while giving you the tools you need to debug effectively.

Using kubectl debug

The kubectl debug command is a powerful tool that simplifies the creation of ephemeral containers. Unlike kubectl exec, which logs into the existing container, kubectl debug creates a new container within the same namespace. This container can run a different OS, mount the application container’s filesystem, and provide all necessary debugging tools without altering the application’s state. This method ensures you can inspect and diagnose issues even if the original container is not operational.

For example, let’s consider a scenario where we’re debugging a container using an ephemeral Ubuntu container:

kubectl debug <myapp> -it <pod-name> --image=ubuntu --share-process --copy-to=<myapp-debug>

This command launches a new Ubuntu-based container within the same pod, providing a full-fledged environment to diagnose the application container. Even if the original container lacks a shell or crashes, the ephemeral container remains operational, allowing you to perform necessary checks and install tools as needed. It relies on the fact that we can have multiple containers in the same pod, that way we can inspect the filesystem of the debugged container without physically entering that container.

Practical Application of Ephemeral Containers

To illustrate, let’s delve deeper into how ephemeral containers can be used in real-world scenarios. Suppose you have a container that consistently crashes due to a mysterious issue. By deploying an ephemeral container with a comprehensive set of debugging tools, you can monitor the logs, inspect the filesystem, and trace processes without worrying about the constraints of the original container environment.

For instance, you might encounter a situation where an application container crashes due to an unhandled exception. By using kubectl debug, you can create an ephemeral container that shares the same network namespace as the original container. This allows you to capture network traffic and analyze it to understand if there are any issues related to connectivity or data corruption.

Security Considerations

While ephemeral containers reduce the risk of impacting the production environment, they still pose security risks. It’s critical to restrict access to debugging tools and ensure that only authorized personnel can deploy ephemeral containers. Treat access to these systems with the same caution as handing over the keys to your infrastructure.

Ephemeral containers, by their nature, can access sensitive information within the pod. Therefore, it is essential to enforce strict access controls and audit logs to track who is deploying these containers and what actions are being taken. This ensures that the debugging process does not introduce new vulnerabilities or expose sensitive data.

Interlude: The Role of Observability

While tools like kubectl exec and kubectl debug are invaluable for troubleshooting, they are not replacements for comprehensive observability solutions. Observability allows you to monitor, trace, and log the behavior of your applications in real-time, providing deeper insights into issues without the need for intrusive debugging sessions.

These tools aren't meant for everyday debugging, that role should be occupied by various observability tools. I will discuss observability in more detail in an upcoming post.

Command Line Debugging

While tools like kubectl exec and kubectl debug are invaluable, there are times when you need to dive deep into the application code itself. This is where we can use command line debuggers. Command line debuggers allow you to inspect the state of your application at a very granular level, stepping through code, setting breakpoints, and examining variable states. Personally, I don't use them much

For instance, Java developers can use jdb, the Java Debugger, which is analogous to gdb for C/C++ programs. Here’s a basic rundown of how you might use jdb in a Kubernetes environment:

Set Up Debugging: First, you need to start your Java application with debugging enabled. This typically involves adding a debug flag to your Java command. However, as discussed in my post here, there's an even more powerful way that doesn't require a restart:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar myapp.jar

Port Forwarding: Since the debugger needs to connect to the application, you’ll set up port forwarding to expose the debug port of your pod to your local machine. This is important as JDWP is dangerous:

kubectl port-forward <pod-name> 5005:5005

Connecting the Debugger: With port forwarding in place, you can now connect jdb to the remote application:

jdb -attach localhost:5005

From here, you can use jdb commands to set breakpoints, step through code, and inspect variables. This process allows you to debug issues within the code itself, which can be invaluable for diagnosing complex problems that aren’t immediately apparent through logs or superficial inspection.

Connecting a Standard IDE for Remote Debugging

I prefer IDE debugging by far. I never used JDB for anything other than a demo. Modern IDEs support remote debugging, and by leveraging Kubernetes port forwarding, you can connect your IDE directly to a running application inside a pod.

To set up remote debugging we start with the same steps as the command line debugging. Configuring the application and setting up the port forwarding.

1. Configure the IDE: In your IDE (e.g., IntelliJ IDEA, Eclipse), set up a remote debugging configuration. Specify the host as localhost and the port as 5005.

2. Start Debugging: Launch the remote debugging session in your IDE. You can now set breakpoints, step through code, and inspect variables directly within the IDE, just as if you were debugging a local application.

I show how to do it in IntelliJ/IDEA here.

Conclusion

Debugging Kubernetes environments requires a blend of traditional techniques and modern tools designed for container orchestration. Understanding the limitations of kubectl exec and the benefits of ephemeral containers can significantly enhance your troubleshooting process. However, the ultimate goal should be to build robust observability into your applications, reducing the need for ad-hoc debugging and enabling proactive issue detection and resolution.

By following these guidelines and leveraging the right tools, you can navigate the complexities of Kubernetes debugging with confidence and precision. In the next installment of this series, we’ll delve into common configuration issues in Kubernetes and how to address them effectively.

This part of the debugging series is designed for developers looking to effectively tackle application issues within Kubernetes environments, without delving deeply into the complex DevOps aspects typically associated with its operations. Kubernetes is a big subject, it took me two videos just to explain the basic concepts and background.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Introduction to Kubernetes and Distributed Systems

Kubernetes, while often discussed in the context of cloud computing and large-scale operations, is not just a tool for managing containers. Its principles apply broadly to all large-scale distributed systems. In this post I want to explore Kubernetes from the ground up, emphasizing its role in solving real-world problems faced by developers in production environments.

The Evolution of Deployment Technologies

Before Kubernetes, the deployment landscape was markedly different. Understanding this evolution helps appreciate the challenges Kubernetes aims to solve. The image below represents the road to Kubernetes and the technologies we passed along the way.

In the image we can see that initially, applications were deployed directly onto physical servers. This process was manual, error-prone, and difficult to replicate across multiple environments. For instance, if a company needed to scale its application, it involved procuring new hardware, installing operating systems, and configuring the application from scratch. This could take weeks or even months, leading to significant downtime and operational inefficiencies.

Imagine a retail company preparing for the holiday season surge. Each time they needed to handle increased traffic, they would manually set up additional servers. This was not only time-consuming but also prone to human error. Scaling down after the peak period was equally cumbersome, leading to wasted resources.

Enter Virtualization

Virtualization technology introduced a layer that emulated the hardware, allowing for easier replication and migration of environments but at the cost of performance. However, fast virtualization enabled the cloud revolution. It let companies like Amazon lease its servers at scale without compromising their own workloads.

Virtualization involves running multiple operating systems on a single physical hardware host. Each virtual machine (VM) includes a full copy of an operating system, the application, necessary binaries, and libraries—taking up tens of GBs. VMs are managed via a hypervisor, such as VMware's ESXi or Microsoft's Hyper-V, which sits between the hardware and the operating system and is responsible for distributing hardware resources among the VMs. This layer adds additional overhead and can lead to decreased performance due to the need to emulate hardware.

Note that virtualization is often referred to as "virtual machines", I chose to avoid that terminology due to the focus of this blog on Java and the JVM where a virtual machine is typically a reference to the Java Virtual Machine (JVM).

Rise of Containers

Containers emerged as a lightweight alternative to full virtualization. Tools like Docker standardized container formats, making it easier to create and manage containers without the overhead associated with traditional virtual machines. Containers encapsulate an application’s runtime environment, making them portable and efficient.

Unlike virtualization, containerization encapsulates an application in a container with its own operating environment, but it shares the host system’s kernel with other containers. Containers are thus much more lightweight, as they do not require a full OS instance; instead, they include only the application and its dependencies, such as libraries and binaries. This setup reduces the size of each container and improves boot times and performance by removing the hypervisor layer.

Containers operate using several key Linux kernel features:

• Namespaces: Containers use namespaces to provide isolation for global system resources between independent containers. This includes aspects of the system like process IDs, networking interfaces, and file system mounts. Each container has its own isolated namespace, which gives it a private view of the operating system with access only to its resources.

• Control Groups (cgroups): Cgroups further enhance the functionality of containers by limiting and prioritizing the hardware resources a container can use. This includes parameters such as CPU time, system memory, network bandwidth, or combinations of these resources. By controlling resource allocation, cgroups ensure that containers do not interfere with each other’s performance and maintain the efficiency of the underlying server.

• Union File Systems: Containers use union file systems, such as OverlayFS, to layer files and directories in a lightweight and efficient manner. This system allows containers to appear as though they are running on their own operating system and file system, while they are actually sharing the host system’s kernel and base OS image.

Rise of Orchestration

As containers began to replace virtualization due to their efficiency and speed, developers and organizations rapidly adopted them for a wide range of applications. However, this surge in container usage brought with it a new set of challenges, primarily related to managing large numbers of containers at scale.

While containers are incredibly efficient and portable, they introduce complexities when used extensively, particularly in large-scale, dynamic environments:

• Management Overhead: Manually managing hundreds or even thousands of containers quickly becomes unfeasible. This includes deployment, networking, scaling, and ensuring availability and security.

• Resource Allocation: Containers must be efficiently scheduled and managed to optimally use physical resources, avoiding underutilization or overloading of host machines.

• Service Discovery and Load Balancing: As the number of containers grows, keeping track of which container offers which service and how to balance the load between them becomes critical.

• Updates and Rollbacks: Implementing rolling updates, managing version control, and handling rollbacks in a containerized environment require robust automation tools.

To address these challenges, the concept of container orchestration was developed. Orchestration automates the scheduling, deployment, scaling, networking, and lifecycle management of containers, which are often organized into microservices. Efficient orchestration tools help ensure that the entire container ecosystem is healthy and that applications are running as expected.

Enter Kubernetes

Among the orchestration tools, Kubernetes emerged as a frontrunner due to its robust capabilities, flexibility, and strong community support. Kubernetes offers several features that address the core challenges of managing containers:

• Automated Scheduling: Kubernetes intelligently schedules containers on the cluster’s nodes, taking into account the resource requirements and other constraints, optimizing for efficiency and fault tolerance.

• Self-Healing Capabilities: It automatically replaces or restarts containers that fail, ensuring high availability of services.

• Horizontal Scaling: Kubernetes can automatically scale applications up and down based on demand, which is essential for handling varying loads efficiently.

• Service Discovery and Load Balancing: Kubernetes can expose a container using the DNS name or using its own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.

• Automated Rollouts and Rollbacks: Kubernetes allows you to describe the desired state for your deployed containers using declarative configuration, and can change the actual state to the desired state at a controlled rate, such as to roll out a new version of an application.

Why Kubernetes Stands Out

Kubernetes not only solves practical, operational problems associated with running containers but also integrates with the broader technology ecosystem, supporting continuous integration and continuous deployment (CI/CD) practices. It is backed by the Cloud Native Computing Foundation (CNCF), ensuring it remains cutting-edge and community-focused.

There used to be a site called "doyouneedkubernetes.com" when you visited that site it said "No". Most of us don't need Kubernetes and it is often a symptom of Resume Driven Design (RDD). However, even when we don't need its scaling capabilities the advantages of its standardization are tremendous. Kubernetes became the de-facto standard and created a cottage industry of tools around it. Features such as, observability and security can be plugged in easily. Cloud migration becomes arguably easier. Kubernetes is now the "lingua franca" of production environments.

Kubernetes For Developers

Understanding Kubernetes architecture is crucial for debugging and troubleshooting. The following image shows the high level view of a Kubernetes deployment. There are far more details in most tutorials geared towards DevOps engineers, but for a developer the point that matters is just "Your Code": that tiny corner at the edge.

In the image above we can see:

• Master Node (represented by the blue Kubernetes logo on the left): The control plane of Kubernetes, responsible for managing the state of the cluster, scheduling applications, and handling replication.

• Worker Nodes: These nodes contain the pods that run the containerized applications. Each worker node is managed by the master.

• Pods: The smallest deployable units created and managed by Kubernetes, usually containing one or more containers that need to work together.

These components work together to ensure that an application runs smoothly and efficiently across the cluster.

Kubernetes Basics In Practice

Up until now this post has been theory heavy, let's review some commands we can use to work with a Kubernetes cluster. First we would want to list the pods we have within the cluster which we can do using the get pods command as such:

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
  my-first-pod-id-xxxx     1/1    Running   0          13s
  my-second-pod-id-xxxx    1/1    Running   0          13s

A command such as kubectl describe pod returns high level description of the pod such as its name, parent node, etc. Many problems in production pods can be solved by looking at the system log, this can be accomplished by invoking the logs command:

$ kubectl logs -f <pod>
[2022-11-29 04:12:17,262] INFO log data
...

Most typical large scale applications logs are ingested by tools such as Elastic, Loki etc. As such, the logs command isn't as useful in production except for debugging edge cases.

Final Word

This introduction to Kubernetes has set the stage for deeper exploration into specific debugging and troubleshooting techniques, which we will cover in the upcoming posts. The complexity of Kubernetes makes is much harder to debug, but there are facilities in place to workaround some of that complexity.

While this article (and its followups) focus on Kubernetes, future posts will delve into observability and related tools, which are crucial for effective debugging in production environments.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Fail-Fast

Fail-fast systems are designed to immediately stop functioning upon encountering an unexpected condition. This immediate failure helps to catch errors early, making debugging more straightforward.

The fail-fast approach ensures that errors are caught immediately. For example, in the world of programming languages, Java embodies this approach by producing a NullPointerException instantly when encountering a null value, stopping the system and making the error clear. This immediate response helps developers identify and address issues quickly, preventing them from becoming more serious.

By catching and stopping errors early, fail-fast systems reduce the risk of cascading failures, where one error leads to others. This makes it easier to contain and resolve issues before they spread through the system, preserving overall stability.

It is easy to write unit and integration tests for fail-fast systems. This advantage is even more pronounced when we need to understand the test failure. Fail-fast systems usually point directly at the problem in the error stack trace.

However, fail-fast systems carry their own risks, particularly in production environments:

• Production Disruptions: If a bug reaches production, it can cause immediate and significant disruptions, potentially impacting both system performance and the business’s operations.

• Risk Appetite: Fail-fast systems require a level of risk tolerance from both engineers and executives. They need to be prepared to handle and address failures quickly, often balancing this with potential business impacts.

Fail-Safe

Fail-safe systems take a different approach, aiming to recover and continue even in the face of unexpected conditions. This makes them particularly suited for uncertain or volatile environments.

Microservices are a prime example of fail-safe systems, embracing resiliency through their architecture. Circuit breakers, both physical and software-based, disconnect failing functionality to prevent cascading failures, helping the system continue operating.

Fail-safe systems ensure that systems can survive even harsh production environments, reducing the risk of catastrophic failure. This makes them particularly suited for mission-critical applications, such as in hardware devices or aerospace systems, where smooth recovery from errors is crucial.

However, fail-safe systems have downsides:

• Hidden Errors: By attempting to recover from errors, fail-safe systems can delay the detection of issues, making them harder to trace and potentially leading to more severe cascading failures.

• Debugging Challenges: This delayed nature of errors can complicate debugging, requiring more time and effort to find and resolve issues.

Choosing Between Fail-Fast and Fail-Safe

It's challenging to determine which approach is better, as both have their merits. Fail-fast systems offer immediate debugging, lower risk of cascading failures, and quicker detection and resolution of bugs. This helps catch and fix issues early, preventing them from spreading.

Fail-safe systems handle errors gracefully, making them better suited for mission-critical systems and volatile environments, where catastrophic failures can be devastating.

Balancing Both

To leverage the strengths of each approach, a balanced strategy can be effective:

• Fail-Fast for Local Services: When invoking local services like databases, fail-fast can catch errors early, preventing cascading failures.

• Fail-Safe for Remote Resources: When relying on remote resources, such as external web services, fail-safe can prevent disruptions from external failures.

A balanced approach also requires clear and consistent implementation throughout coding, reviews, tooling, and testing processes, ensuring it is integrated seamlessly. Fail-fast can integrate well with orchestration and observability. Effectively, this moves the fail-safe aspect to a different layer of OPS instead of into the developer layer.

Consistent Layer Behavior

This is where things get interesting. It isn't about choosing between fail-safe and fail-fast. It's about choosing the right layer for them. E.g. if an error is handled in a deep layer using a fail-safe approach, it won't be noticed. This might be OK, but if that error has adverse impact (performance, garbage data, corruption, security, etc.) then we will have a problem later on and won't have a clue.

The right solution is to handle all errors in a single layer, in modern systems the top layer is the OPS layer and it makes the most sense. It can report the error to the engineers who are most qualified to deal with the error. But they can also provide immediate mitigation such as restarting a service, allocating additional resources or reverting a version.

Retry's Are not Fail-Safe

Recently I was at a lecture where the speakers listed their updated cloud architecture. They chose to take a shortcut to microservices by using a framework that allows them to retry in the case of failure. Unfortunately, failure doesn't behave the way we would like. You can't eliminate it completely through testing alone. Retry, isn't fail-safe. In-fact: it can mean catastrophe.

They tested their system and "it works", even in production. But lets assume that a catastrophic situation does occur, their retry mechanism can operate as a denial of service attack against their own servers. The number of ways in which ad-hoc architectures such as this can fail are mind-boggling.

This is especially important once we redefine failures.

Redefining Failure

Failures in software systems aren't just about crashes. A crash can be seen as a simple and immediate failure, but there are more complex issues to consider. In fact, crashes in the age of containers are probably the best failures. A system restarts seamlessly with barely an interruption.

Data Corruption

Data corruption is far more severe and insidious than a crash. It carries with it long-term consequences. Corrupted data can lead to security and reliability problems that are challenging to fix, requiring extensive reworking and potentially unrecoverable data.

Cloud computing has led to defensive programming techniques, like circuit breakers and retries, emphasizing comprehensive testing and logging to catch and handle failures gracefully. In a way, this environment sent us back in terms of quality.

A fail-fast system at the data level could stop this from happening. Addressing a bug goes beyond a simple fix. It requires understanding its root cause and preventing reoccurrence, extending into comprehensive logging, testing, and process improvements. This ensures that the bug is fully addressed, reducing the chances of it reoccurring.

Don't Fix the Bug

If it's a bug in production you should probably revert, if you can't instantly revert production. This should always be possible and if it isn't this is something you should work on.

Failures must be fully understood before a fix is undertaken. In my own companies I often skipped that step due to pressure, in a small startup that is forgivable. In larger companies we need to understand the root cause. A culture of debrief for bugs and production issues is essential. The fix should also include process mitigation that prevents similar issues from reaching production.

Debugging Failure

Fail-fast systems are much easier to debug. They have inherently simpler architecture and it is easier to pinpoint an issue to a specific area. It is crucial to throw exceptions even for minor violations (e.g. validations). This prevents cascading types of bugs that prevail in loose systems.

This should be further enforced by unit tests that verify the limits we define and verify proper exceptions are thrown. Retries should be avoided in the code as they make debugging exceptionally difficult and their proper place is in the OPS layer. To facilitate that further, timeouts should be short by default.

Avoiding Cascading Failure

Failure isn't something we can avoid, predict or fully test against. The only thing we can do is soften the blow when a failure occurs. Often this "softening" is achieved by using long running tests meant to replicate extreme conditions as much as possible with the goal of finding our applications weak spots. This is rarely enough, robust systems need to revise these tests often based on real production failures.

A great example of fail-safe would be a cache of REST responses that lets us keep working even when a service is down. Unfortunately, this can lead to complex niche issues such as cache poisoning or a situation in which a banned user still had access due to cache.

Hybrid in Production

Fail-safe is best applied only in production/staging and in the OPS layer. This reduces the amount of changes between production and dev, we want them to be as similar as possible, yet it's still a change which can negatively impact production. But the benefits are tremendous as observability can get a clear picture of system failures.

The discussion here is a bit colored by my more recent experience of building observable cloud architectures. However, the same principle applies to any type of software whether embedded or in the cloud. In such cases we often choose to implement fail-safe in the code, in this case I would suggest implementing it consistently and consciously in a specific layer.

There's also a special case of libraries/frameworks that often provide inconsistent and badly documented behaviors in these situations. I myself am guilty of such inconsistency in some of my work. It's an easy mistake to make.

Final Word

This is my last post on the theory of debugging series that's part of my book/course on debugging. We often think of debugging as the action we take when something fails, it isn't. Debugging starts the moment we write the first line of code. We make decisions that will impact the debugging process as we code, often we're just unaware of these decisions until we get a failure.

I hope this post and series will help you write code that is prepared for the unknown. Debugging, by its nature, deals with the unexpected. Tests can't help. But as I illustrated in my previous posts, there are many simple practices we can undertake that would make it easier to prepare. This isn't a one time process, it's an iterative process that requires re-evaluation of decisions made as we encounter failure.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

The Intersection of Debugging and Testing

Debugging and testing play distinct roles in software development. Debugging is the targeted process of identifying and fixing known bugs. Testing, on the other hand, encompasses a adjacent scope, identifying unknown issues by validating expected software behavior across a variety of scenarios.

Both are a part of the debug fix cycle which is a core concept in debugging. Before we cover the cycle we should first make sure we're aligned on the basic terminology.

Unit Tests

Unit tests are tightly linked to debugging efforts, focusing on isolated parts of the application—typically individual functions or methods. Their purpose is to validate that each unit operates correctly in isolation, making them a swift and efficient tool in the debugging arsenal. These tests are characterized by their speed and consistency, enabling developers to run them frequently, sometimes even automatically as code is written within the IDE.

Since software is so tightly bound it is nearly impossible to compose unit tests without extensive mocking. Mocking involves substituting a genuine component with a stand-in that returns predefined results, thus a test method can simulate scenarios without relying on the actual object. This is a powerful yet controversial tool. By using mocking we're in-effect creating a synthetic environment that might misrepresent the real world. We're reducing the scope of the test and might perpetuate some bugs.

Integration Tests

Opposite to unit tests, integration tests examine the interactions between multiple units, providing a more comprehensive picture of the system's health. While they cover broader scenarios, their setup can be more complex due to the interactions involved. However, they are crucial in catching bugs that arise from the interplay between different software components.

In general mocking can be used in integration tests but it is discouraged. They take longer to run and are sometimes harder to set up. However, many developers (myself included) would argue that they are the only benchmark for quality. Most bugs express themselves in the seams between the modules and integration tests are better at detecting that.

Since they are far more important some developers would argue that unit tests are unnecessary. This isn't true, unit test failures are much easier to read and understand. Since they are faster we can run them during development, even while typing. In that sense the balance between the two approaches is the important part.

Coverage

Coverage is a metric that helps quantify the effectiveness of testing by indicating the proportion of code exercised by tests. It helps identify potential areas of the code that have not been tested, which could harbor undetected bugs. However, striving for 100% coverage can be a case of diminishing returns; the focus should remain on the quality and relevance of the tests rather than the metric itself. In my experience, chasing high coverage numbers often results in bad test practices that persist problems.

It is my opinion that unit tests should be excluded from coverage metrics due to the importance of integration tests to overall quality. To get a sense of quality coverage should focus on integration and end to end tests.

The Debug-Fix Cycle

The debug-fix cycle is a structured approach that integrates testing into the debugging process. The stages include identifying the bug, creating a test that reproduces the bug, fixing the bug, verifying the fix with the test, and finally, running the application to ensure the fix works in the live environment. This cycle emphasizes the importance of testing in not only identifying but also in preventing the recurrence of bugs.

Notice that this is a simplified version of the cycle with a focus on the testing aspect only. The full cycle includes discussion of the issue tracking and versioning as part of the whole process. I discuss this more in-depth in other posts in the series and my book.

Composing Tests with Debuggers

A powerful feature of using debuggers in test composition is their ability to "jump to line" or "set value." Developers can effectively reset the execution to a point before the test and rerun it with different conditions, without recompiling or rerunning the entire suite. This iterative process is invaluable for achieving desired test constraints and improves the quality of unit tests by refining the input parameters and expected outcomes.

Increasing test coverage is about more than hitting a percentage; it's about ensuring that tests are meaningful and that they contribute to software quality. A debugger can significantly assist in this by identifying untested paths. When a test coverage tool highlights lines or conditions not reached by current tests, the debugger can be used to force execution down those paths. This helps in crafting additional tests that cover missed scenarios, ensuring that the coverage metric is not just a number but a true reflection of the software's tested state.

In this case you will notice that the next line in the body is a rejectValue call which will throw an exception. I don’t want an exception thrown as I still want to test all the permutations of the method. I can drag the execution pointer (arrow on the left) and place it back at the start of the method.

Test-Driven Development

How does all of this fit with disciplines like Test-Driven Development (TDD)?

It doesn't fit well. Before we get into that let's revisit the basics of TDD. Weak TDD typically means just writing tests before writing the code. Strong TDD involves a red-green-refactor cycle:

1. Red: Write a test that fails because the feature it tests isn't implemented yet.

2. Green: Write the minimum amount of code necessary to make the test pass.

3. Refactor: Clean up the code while ensuring that tests continue to pass.

This rigorous cycle guarantees that new code is continually tested and refactored, reducing the likelihood of complex bugs. It also means that when bugs do appear, they are often easier to isolate and fix due to the modular and well-tested nature of the codebase. At least, that's the theory.

TDD can be especially advantageous for scripting and loosely typed languages. In environments lacking the rigid structure of compilers and linters, TDD steps in to provide the necessary checks that would otherwise be performed during compilation in statically typed languages. It becomes a crucial substitute for compiler/linter checks, ensuring that type and logic errors are caught early.

In real-world application development, TDD's utility is nuanced. While it encourages thorough testing and upfront design, it can sometimes hinder the natural flow of development, especially in complex systems that evolve through numerous iterations. The requirement for 100% test coverage can lead to an unnecessary focus on fulfilling metrics rather than writing meaningful tests.

The biggest problem in TDD is its focus on unit testing. TDD is impractical with integration tests as the process would take too long. But as we determined in the start of this post, integration tests are the true benchmark for quality. In that test TDD is a methodology that provides great quality for arbitrary tests, but not necessarily great quality for the final product. You might have the best cog in the world, but if doesn't fit well into the machine then it isn't great.

Final Word

Debugging is a tool that not only fixes bugs but also actively aids in crafting tests that bolster software quality. By utilizing debuggers in test composition and increasing coverage, developers can create a suite of tests that not only identifies existing issues but also guards against future ones, thus ensuring the delivery of reliable, high-quality software.

Debugging lets us increase coverage and verify edge cases effectively. It's part of a standardized process for issue resolution that's critical for reliability and prevents regressions.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Introduction to Wireshark

Wireshark was initially developed by Eric Rescorla and Gerald Combs, designed to capture and analyze network packets in real-time. Its capabilities extend across various network interfaces and protocols, making it a versatile tool for anyone involved in networking. Unlike its command-line counterpart, tcpdump, Wireshark's graphical interface simplifies the analysis process, presenting data in a user-friendly "proto view" that organizes packets in a hierarchical structure. This facilitates quick identification of protocols, ports, and data flows.

The key features of Wireshark are:

• Graphical User Interface (GUI): Eases the analysis of network packets compared to command-line tools.

• Proto View: Displays packet data in a tree structure, simplifying protocol and port identification.

• Compatibility: Supports a wide range of network interfaces and protocols.

Browser Network Monitors

FireFox and Chrome contain a far superior network monitor tool built into them. It is superior because it is simpler to use and works with secure websites out of the box. If you can use the browser to debug the network traffic you should do that.

In the cases where your traffic requires low level protocol information or is outside of the browser, Wireshark is the next best thing.

Installation and Getting Started

To begin with Wireshark, visit their official website for the download. The installation process is straightforward, but attention should be paid to the installation of command-line tools, which may require separate steps. Upon launching Wireshark, users are greeted with a selection of network interfaces as seen below. Choosing the correct interface, such as the loopback for local server debugging, is crucial for capturing relevant data.

When debugging a Local Server (localhost) use the loopback interface. Remote servers will probably fit with the en0 network adapter. You can use the activity graph next to the network adapter to identify active interfaces for capture.

Navigating Through Noise with Filters

One of the challenges of using Wireshark is the overwhelming amount of data captured, including irrelevant "background noise" as seen in the following image.

Wireshark addresses this with powerful display filters, allowing users to hone in on specific ports, protocols, or data types. For instance, filtering TCP traffic on port 8080 can significantly reduce unrelated data, making it easier to debug specific issues.

Notice that the there is a completion widget on top of the Wireshark UI that lets you find out the values more easily.

In this case we filter by port tcp.port == 8080 which is the port used typically in Java servers (e.g. Spring Boot/tomcat).

But this isn't enough as HTTP is more concise. We can filter by protocol by adding http to the filter which narrows the view to HTTP requests and responses as shown in the following image.

Deep Dive into Data Analysis

Wireshark excels in its ability to dissect and present network data in an accessible manner. For example, HTTP responses carrying JSON data are automatically parsed and displayed in a readable tree structure as seen below. This feature is invaluable for developers and analysts, providing insights into the data exchanged between clients and servers without manual decoding.

Wireshark parses and displays JSON data within the packet analysis pane. It offers both hexadecimal and ASCII views for raw packet data.

Beyond Basic Usage

While Wireshark's basic functionalities cater to a wide range of networking tasks, its true strength lies in advanced features such as ethernet network analysis, HTTPS decryption, and debugging across devices. These tasks, however, may involve complex configuration steps and a deeper understanding of network protocols and security measures.

There are two big challenges when working with Wireshark:

• HTTPS Decryption: Decrypting HTTPS traffic requires additional configuration but offers visibility into secure communications.

• Device Debugging: Wireshark can be used to troubleshoot network issues on various devices, requiring specific knowledge of network configurations.

The Basics of HTTPS Encryption

HTTPS uses the Transport Layer Security (TLS) or its predecessor, Secure Sockets Layer (SSL), to encrypt data. This encryption mechanism ensures that any data transferred between the web server and the browser remains confidential and untouched. The process involves a series of steps including handshake, data encryption, and data integrity checks.

Decrypting HTTPS traffic is often necessary for developers and network administrators to troubleshoot communication errors, analyze application performance, or ensure that sensitive data is correctly encrypted before transmission. It's a powerful capability in diagnosing complex issues that cannot be resolved by simply inspecting unencrypted traffic or server logs.

Methods for Decrypting HTTPS in Wireshark

Important: Decrypting HTTPS traffic should only be done on networks and systems you own or have explicit permission to analyze. Unauthorized decryption of network traffic can violate privacy laws and ethical standards.

Pre-Master Secret Key Logging

One common method involves using the pre-master secret key to decrypt HTTPS traffic. Browsers like Firefox and Chrome can log the pre-master secret keys to a file when configured to do so. Wireshark can then use this file to decrypt the traffic:

1. Configure the Browser: Set an environment variable (SSLKEYLOGFILE) to specify a file where the browser will save the encryption keys.

2. Capture Traffic: Use Wireshark to capture the traffic as usual.

3. Decrypt the Traffic: Point Wireshark to the file with the pre-master secret keys (through Wireshark's preferences) to decrypt the captured HTTPS traffic.

Using a Proxy

Another approach involves routing traffic through a proxy server that decrypts HTTPS traffic and then re-encrypts it before sending it to the destination. This method might require setting up a dedicated decryption proxy that can handle the TLS encryption/decryption:

1. Set Up a Decryption Proxy: Tools like Mitmproxy or Burp Suite can act as an intermediary that decrypts and logs HTTPS traffic.

2. Configure Network to Route Through Proxy: Ensure the client's network settings route traffic through the proxy.

3. Inspect Traffic: Use the proxy's tools to inspect the decrypted traffic directly.

Integrating tcpdump with Wireshark for Enhanced Network Analysis

While Wireshark offers a graphical interface for analyzing network packets, there are scenarios where using it directly may not be feasible due to security policies or operational constraints. tcpdump, a powerful command-line packet analyzer, becomes invaluable in these situations, providing a flexible and less intrusive means of capturing network traffic.

The Role of tcpdump in Network Troubleshooting

tcpdump allows for the capture of network packets without a graphical user interface, making it ideal for use in environments with strict security requirements or limited resources. It operates under the principle of capturing network traffic to a file, which can then be analyzed at a later time or on a different machine using Wireshark.

Key Scenarios for tcpdump Usage:

• High-security Environments: In places like banks or government institutions where running network sniffers might pose a security risk, tcpdump offers a less intrusive alternative.

• Remote Servers: Debugging issues on a cloud server can be challenging with Wireshark due to the graphical interface; tcpdump captures can be transferred and analyzed locally.

• Security-conscious Customers: Customers may be hesitant to allow third-party tools to run on their systems; tcpdump's command-line operation is often more palatable.

Using tcpdump Effectively

Capturing traffic with tcpdump involves specifying the network interface and an output file for the capture. This process is straightforward but powerful, allowing for detailed analysis of network interactions:

1. Command Syntax: The basic command structure for initiating a capture involves specifying the network interface (e.g., en0 for wireless connections) and the output file name.

2. Execution: Once the command is run, tcpdump silently captures network packets. The capture continues until it's manually stopped, at which point the captured data can be saved to the specified file.

3. Opening Captures in Wireshark: The file generated by tcpdump can be opened in Wireshark for detailed analysis, utilizing Wireshark's advanced features for dissecting and understanding network traffic.

The following shows the tcpdump command and its output:

$ sudo tcpdump -i en0 -w output
Password:
tcpdump: listening on en, link-type EN10MB (Ethernet), capture size 262144 bytes
^C3845 packets captured
4189 packets received by filter
0 packets dropped by kernel

Challenges and Considerations

Identifying the correct network interface for capture on remote systems might require additional steps, such as using the ifconfig command to list available interfaces. This step is crucial for ensuring that relevant traffic is captured for analysis.

Final Word

Wireshark stands out as a powerful tool for network analysis, offering deep insights into network traffic and protocols. Whether it's for low-level networking work, security analysis, or application development, Wireshark's features and capabilities make it an essential tool in the tech arsenal. With practice and exploration, users can leverage Wireshark to uncover detailed information about their networks, troubleshoot complex issues, and secure their environments more effectively.

Wireshark's blend of ease of use with profound analytical depth ensures it remains a go-to solution for networking professionals across the spectrum. Its continuous development and wide-ranging applicability underscore its position as a cornerstone in the field of network analysis.

Combining tcpdump's capabilities for capturing network traffic with Wireshark's analytical prowess offers a comprehensive solution for network troubleshooting and analysis. This combination is particularly useful in environments where direct use of Wireshark is not possible or ideal. While both tools possess a steep learning curve due to their powerful and complex features, they collectively form an indispensable toolkit for network administrators, security professionals, and developers alike.

This integrated approach not only addresses the challenges of capturing and analyzing network traffic in various operational contexts but also highlights the versatility and depth of tools available for understanding and securing modern networks.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Introduction

Java 9 introduced many changes, with modules as the highlight. However, among these significant shifts, jhsdb didn’t get the attention it deserved. Officially, Oracle describes jhsdb as a Serviceability Agent tool, part of the JDK aimed at snapshot debugging, performance analysis, and offering deep insights into the Hotspot JVM and Java applications running on it. Simply put, jhsdb is your go-to for delving into JVM internals, understanding core dumps, and diagnosing JVM or native library failures.

Getting Started with jhsdb

To begin we can invoke:

$ jhsdb --help
clhsdb           command line debugger
hsdb             ui debugger
debugd --help    to get more information
jstack --help    to get more information
jmap   --help    to get more information
jinfo  --help    to get more information
jsnap  --help    to get more information

This command in reveals that jhsdb includes six distinct tools:

1. debugd: A remote debug server for connecting and diagnosing remotely.

2. jstack: Provides detailed stack and lock information.

3. jmap: Offers insights into heap memory.

4. jinfo: Displays basic JVM information.

5. jsnap: Assists with performance data.

6. Command Line Debugger: Although there's a preference for the GUI, we'll focus on GUI Debugging for a more visual approach.

Let's dive into these tools and explore how they can aid in diagnosing and resolving JVM issues.

Understanding and Using debugd

debugd might not be your first choice for production environments due to its remote debugging nature. Yet, it could be valuable for local container debugging. To use it we first need to detect the JVM process ID (PID) which we can accomplish using the jps command. Unfortunately, because of a bug in the UI you can’t currently connect to a remote server via the GUI debugger. I could only use this with command-line tools such as jstack (discussed below).

With the command:

jhsdb debugd --pid 1234

We can connect to the process 1234. We can then use a tool like jstack to get additional information:

jhsdb jstack --connect localhost

Notice that the --connect argument applies globally and should work for all commands.

Leveraging jstack for Thread Dumps

jstack is instrumental in generating thread dumps, crucial for analyzing stack processes in user machines or production environments. This command can reveal detailed JVM running states, including deadlock detection, thread statuses, and compilation insights.

Typically we would use jstack locally which removes the need for debugd:

$ jhsdb jstack --pid 1234
Attaching to process ID 1234, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.13+8-LTS
Deadlock Detection:

No deadlocks found.

"Keep-Alive-Timer" #189 daemon prio=8 tid=0x000000011d81f000 nid=0x881f waiting on condition [0x0000000172442000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   JavaThread state: _thread_blocked
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - sun.net.www.http.KeepAliveCache.run() @bci=3, line=168 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=829 (Interpreted frame)
 - jdk.internal.misc.InnocuousThread.run() @bci=20, line=134 (Interpreted frame)


"DestroyJavaVM" #171 prio=5 tid=0x000000011f809000 nid=0x2703 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_blocked

This snapshot can help us infer many details about how the application acts locally and in production.

Is our code compiled?

Is it waiting on a monitor?

What other threads are running and what are they doing?

Heap Memory Analysis with jmap

For a deep dive into RAM and heap memory, jmap is unmatched. It displays comprehensive heap memory details, aiding in GC tuning and performance optimization. Particularly useful is the histo flag for identifying potential memory leaks through a histogram of RAM usage.

Typical usage of jmap is very similar to jstack and other tools mentioned in this post:

$ jhsdb jmap --pid 1234 --heap
Attaching to process ID 1234, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.13+8-LTS

using thread-local object allocation.
Garbage-First (G1) GC with 9 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 17179869184 (16384.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 10305404928 (9828.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 4194304 (4.0MB)

Heap Usage:
G1 Heap:
   regions  = 4096
   capacity = 17179869184 (16384.0MB)
   used     = 323663048 (308.6691360473633MB)
   free     = 16856206136 (16075.330863952637MB)
   1.8839668948203325% used
G1 Young Generation:
Eden Space:
   regions  = 66
   capacity = 780140544 (744.0MB)
   used     = 276824064 (264.0MB)
   free     = 503316480 (480.0MB)
   35.483870967741936% used
Survivor Space:
   regions  = 8
   capacity = 33554432 (32.0MB)
   used     = 33554432 (32.0MB)
   free     = 0 (0.0MB)
   100.0% used
G1 Old Generation:
   regions  = 4
   capacity = 478150656 (456.0MB)
   used     = 13284552 (12.669136047363281MB)
   free     = 464866104 (443.3308639526367MB)
   2.7783193086322986% used

In most cases this might seem like gibberish but when we experience GC thrashing this might be a secret weapon in your arsenal. You can use this to fine tune GC settings and determine the right parameters to set. Since this can easily run in production you can base this on real world observations.

If you could reproduce a memory leak but you don’t have a debugger attached, you can use this to generate a memory histogram:

$ jhsdb jmap --pid 1234 --histo
Attaching to process ID 72640, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.13+8-LTS
Iterating over heap. This may take a while...
Object Histogram:

num       #instances    #bytes    Class description
--------------------------------------------------------------------------
1:        225689    204096416    int[]
2:        485992    59393024    byte[]
3:        17221    23558328    sun.security.ssl.CipherSuite[]
4:        341376    10924032    java.util.HashMap$Node
5:        117706    9549752    java.util.HashMap$Node[]
6:        306720    7361280    java.lang.String
7:        12718    6713944    char[]
8:        113884    5466432    java.util.HashMap
9:        64683    4657176    java.util.regex.Matcher
10:        95612    4615720    java.lang.Object[]
11:        106233    4249320    java.util.HashMap$KeyIterator
12:        16166    4090488    long[]
13:        126977    4063264    java.util.concurrent.ConcurrentHashMap$Node
14:        150789    3618936    java.util.ArrayList
15:        130167    3546016    java.lang.String[]
16:        156237    3227152    java.lang.Class[]
17:        33145    2916760    java.lang.reflect.Method
18:        32193    2575440    nonapi.io.github.classgraph.fastzipfilereader.FastZipEntry
19:        17314    2051672    java.lang.Class
20:        32043    1794408    io.github.classgraph.ClasspathElementZip$1
21:        107918    1726688    java.util.HashSet
22:        105970    1695520    java.util.HashMap$KeySet

This can help narrow down the source of the issue. There are better tools for that in the IDE and during development. But if you're running a server even locally, it can instantly give you a snapshot of RAM.

Basic JVM Insights with jinfo

Though not as detailed as other commands, jinfo is useful for a quick glance at system properties and JVM flags, especially on unfamiliar machines. It's a straightforward tool that requires just a PID to function.

jhsdb jinfo --pid 1234

Performance Metrics with jsnap

jsnap offers a wealth of internal metrics and statistics, such as thread counts and peak numbers. This data is vital for fine-tuning aspects like thread pool sizes, directly impacting production overhead.

$ jhsdb jsnap --pid 72640
Attaching to process ID 72640, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.13+8-LTS
java.threads.started=418 event(s)
java.threads.live=12
java.threads.livePeak=30
java.threads.daemon=8
java.cls.loadedClasses=16108 event(s)
java.cls.unloadedClasses=0 event(s)
java.cls.sharedLoadedClasses=0 event(s)
java.cls.sharedUnloadedClasses=0 event(s)
java.ci.totalTime=23090159603 tick(s)
java.property.java.vm.specification.version=11
java.property.java.vm.specification.name=Java Virtual Machine Specification
java.property.java.vm.specification.vendor=Oracle Corporation
java.property.java.vm.version=11.0.13+8-LTS
java.property.java.vm.name=OpenJDK 64-Bit Server VM
java.property.java.vm.vendor=Azul Systems, Inc.
java.property.java.vm.info=mixed mode
java.property.jdk.debug=release

GUI Debugging: A Visual Approach

We'll skip over the CLI debugger, the GUI debugger deserves a mention for its user-friendly interface, allowing connections to core files, servers, or PIDs with ease. This visual tool opens up a new dimension in debugging, especially when working with JNI native code.

The GUI debugger can be launched just like any other of the jhsdb tools:

jhsdb hsdb --pid 1234

The GUI layout is designed for ease of navigation, offering a comprehensive view of JVM internals at a glance. Here are some key features and how to use them:

• File Menu: This is your starting point for connecting to debugging targets. You can load core files for post-mortem analysis, attach to running processes to diagnose live issues, or connect to remote debug servers if you’re dealing with distributed systems.

• Threads and Monitors: The GUI provides a real-time view of thread states, making it easier to identify deadlocks, thread contention, and monitor locks. This visual representation simplifies the process of pinpointing concurrency issues that could be affecting application performance.

• Heap Summary: For memory analysis, the GUI debugger gives a graphical overview of heap usage, including generations (for GC analysis), object counts, and memory footprints. This makes identifying memory leaks and optimizing garbage collection strategies more intuitive.

• Method and Stack Inspection: Delving into method executions and stack frames is seamless, allowing you to trace the execution path, inspect local variables, and evaluate the state of the application at different points in time.

Final Word

jhsdb stands out as an essential tool in the debugging toolkit, especially for those dealing with JVM and native code issues. Its range of capabilities, from deep memory analysis to performance metrics, makes it a versatile choice for developers and system administrators alike.

The biggest benefit is in debugging the interaction between Java code and native code. Such code often fails in odd ways and on end user machines. In such situations a typical debugger might not be the best tool and might not expose the whole picture. This is especially true if you get a JVM core dump which is the main use case for jhsdb.

Understanding Java Streams

Java Streams represent a significant shift in how Java developers work with collections and data processing, introducing a functional approach to handling sequences of elements. Streams facilitate declarative processing of collections, enabling operations such as filter, map, reduce, and more in a fluent style. This not only makes the code more readable but also more concise compared to traditional iterative approaches.

A Simple Stream Example

To illustrate, consider the task of filtering a list of names to only include those that start with the letter "J" and then transforming each name into uppercase. Using the traditional approach, this might involve a loop and some if statements. However, with streams, this can be accomplished in a few lines:

List<String> names = Arrays.asList("John", "Jacob", "Edward", "Emily");
// Convert list to stream
List<String> filteredNames = names.stream()       
                  // Filter names that start with "J"
                  .filter(name -> name.startsWith("J"))  
                  // Convert each name to uppercase
                  .map(String::toUpperCase)              
                  // Collect results into a new list
                  .collect(Collectors.toList());         
System.out.println(filteredNames);

Output:

[JOHN, JACOB]

This example demonstrates the power of Java streams: by chaining operations together, we can achieve complex data transformations and filtering with minimal, readable code. It showcases the declarative nature of streams, where we describe what we want to achieve rather than detailing the steps to get there.

What is the peek() Method?

At its core, peek() is a method provided by the Stream interface, allowing developers a glance into the elements of a stream without disrupting the flow of its operations. The signature of peek() is as follows:

Stream<T> peek(Consumer<? super T> action)

It accepts a Consumer functional interface, which means it performs an action on each element of the stream without altering them. The most common use case for peek() is logging the elements of a stream to understand the state of data at various points in the stream pipeline. To understand peek lets look at a sample similar to the previous one:

List<String> collected = Stream.of("apple", "banana", "cherry")
                               .filter(s -> s.startsWith("a"))
                               .collect(Collectors.toList());
System.out.println(collected);

This code filters a list of strings, keeping only the ones that start with "a". While it's straightforward, understanding what happens during the filter operation is not visible.

Debugging with peek()

Now, let's incorporate peek() to gain visibility into the stream:

List<String> collected = Stream.of("apple", "banana", "cherry")
                               .peek(System.out::println) // Logs all elements
                               .filter(s -> s.startsWith("a"))
                               .peek(System.out::println) // Logs filtered elements
                               .collect(Collectors.toList());
System.out.println(collected);

By adding peek() both before and after the filter operation, we can see which elements are processed and how the filter impacts the stream. This visibility is invaluable for debugging, especially when the logic within the stream operations becomes complex.

We can't step over stream operations with the debugger, but peek() provides a glance into the code that is normally obscured from us.

Uncovering Common Bugs with peek()

Filtering Issues

Consider a scenario where a filter condition is not working as expected:

List<String> collected = Stream.of("apple", "banana", "cherry", "Avocado")
                               .filter(s -> s.startsWith("a"))
                               .collect(Collectors.toList());
System.out.println(collected);

Expected output might be ["apple"], but let's say we also wanted "Avocado" due to a misunderstanding of the startsWith method's behavior. Since "Avocado" is spelled with an upper case "A" this code will return false: Avocado".startsWith("a"). Using peek(), we can observe the elements that pass the filter:

List<String> debugged = Stream.of("apple", "banana", "cherry", "Avocado")
                              .peek(System.out::println)
                              .filter(s -> s.startsWith("a"))
                              .peek(System.out::println)
                              .collect(Collectors.toList());
System.out.println(debugged);

Large Data Sets

In scenarios involving large datasets, directly printing every element in the stream to the console for debugging can quickly become impractical. It can clutter the console and make it hard to spot the relevant information. Instead, we can use peek() in a more sophisticated way to selectively collect and analyze data without causing side effects that could alter the behavior of the stream.

Consider a scenario where we're processing a large dataset of transactions, and we want to debug issues related to transactions exceeding a certain threshold:

class Transaction {
    private String id;
    private double amount;

    // Constructor, getters, and setters omitted for brevity
}

List<Transaction> transactions = // Imagine a large list of transactions

// A placeholder for debugging information
List<Transaction> highValueTransactions = new ArrayList<>();

List<Transaction> processedTransactions = transactions.stream()
    // Filter transactions above a threshold
    .filter(t -> t.getAmount() > 5000) 
    .peek(t -> {
        if (t.getAmount() > 10000) {
            // Collect only high-value transactions for debugging
            highValueTransactions.add(t);
        }
     })
     .collect(Collectors.toList());

// Now, we can analyze high-value transactions separately, without overloading the console
System.out.println("High-value transactions count: " + 
       highValueTransactions.size());

In this approach, peek() is used to inspect elements within the stream conditionally. High-value transactions that meet a specific criterion (e.g., amount > 10,000) are collected into a separate list for further analysis. This technique allows for targeted debugging without printing every element to the console, thereby avoiding performance degradation and clutter.

Addressing Side Effects

While streams shouldn't have side effects. In fact, such side effects would break the stream debugger in IntelliJ which I discussed in the past. It's crucial to note that while collecting data for debugging within peek() avoids cluttering the console, it does introduce a side effect to the stream operation, which goes against the recommended use of streams. Streams are designed to be side-effect-free to ensure predictability and reliability, especially in parallel operations.

Therefore, while the above example demonstrates a practical use of peek() for debugging, it's important to use such techniques judiciously. Ideally, this debugging strategy should be temporary and removed once the debugging session is completed to maintain the integrity of the stream's functional paradigm.

Limitations and Pitfalls

While peek() is undeniably a useful tool for debugging Java streams, it comes with its own set of limitations and pitfalls that developers should be aware of. Understanding these can help avoid common traps and ensure that peek() is used effectively and appropriately.

Potential for Misuse in Production Code

One of the primary risks associated with peek() is its potential for misuse in production code. Because peek() is intended for debugging purposes, using it to alter state or perform operations that affect the outcome of the stream can lead to unpredictable behavior. This is especially true in parallel stream operations, where the order of element processing is not guaranteed. Misusing peek() in such contexts can introduce hard-to-find bugs and undermine the declarative nature of stream processing.

Performance Overhead

Another consideration is the performance impact of using peek(). While it might seem innocuous, peek() can introduce a significant overhead, particularly in large or complex streams. This is because every action within peek() is executed for each element in the stream, potentially slowing down the entire pipeline. When used excessively or with complex operations, peek() can degrade performance, making it crucial to use this method judiciously and remove any peek() calls from production code after debugging is complete.

Side Effects and Functional Purity

As highlighted in the enhanced debugging example, peek() can be used to collect data for debugging purposes, but this introduces side effects to what should ideally be a side-effect-free operation. The functional programming paradigm, which streams are a part of, emphasizes purity and immutability. Operations should not alter state outside their scope. By using peek() to modify external state (even for debugging), you're temporarily stepping away from these principles. While this can be acceptable for short-term debugging, it's important to ensure that such uses of peek() do not find their way into production code, as they can compromise the predictability and reliability of your application.

The Right Tool for the Job

Finally, it's essential to recognize that peek() is not always the right tool for every debugging scenario. In some cases, other techniques such as logging within the operations themselves, using breakpoints and inspecting variables in an IDE, or writing unit tests to assert the behavior of stream operations might be more appropriate and effective. Developers should consider peek() as one tool in a broader debugging toolkit, employing it when it makes sense and opting for other strategies when they offer a clearer or more efficient path to identifying and resolving issues.

Navigating the Pitfalls

To navigate these pitfalls effectively:

• Reserve peek() strictly for temporary debugging purposes. If you have a linter as part of your CI tools it might make sense to add a rule that block code from invoking peek().

• Always remove peek() calls from your code before committing it to your codebase, especially for production deployments.

• Be mindful of performance implications and the potential introduction of side effects.

• Consider alternative debugging techniques that might be more suited to your specific needs or the particular issue you're investigating.

By understanding and respecting these limitations and pitfalls, developers can leverage peek() to enhance their debugging practices without falling into common traps or inadvertently introducing problems into their codebases.

Final Thoughts

The peek() method offers a simple yet effective way to gain insights into Java stream operations, making it a valuable tool for debugging complex stream pipelines. By understanding how to use peek() effectively, developers can avoid common pitfalls and ensure their stream operations perform as intended. As with any powerful tool, the key is to use it wisely and in moderation.

The true value of peek() is in debugging massive data sets, these elements are very hard to analyze even with dedicated tools. By using peek() we can dig into said data set and understand the source of the issue programmatically.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

The Need for Advanced Management Tools in Development

Development and DevOps teams utilize an array of tools, often perceived as complex or alien by developers. These tools, designed for scalability, enable the management of thousands of servers simultaneously. Such capabilities, although not always necessary for smaller scales, offer significant advantages in application management. Advanced management tools facilitate the navigation and control over multiple machines, making them indispensable for developers seeking to optimize application performance and reliability.

Introduction to JMX (Java Management Extensions)

One of the pivotal standards in application management is Java Management Extensions (JMX), which Java introduced to simplify the interaction with and management of applications. JMX allows both applications and the Java Development Kit (JDK) itself to expose critical information and functionalities, enabling external tools to manipulate these elements dynamically. Although activating JMX falls outside this discussion, its significance cannot be overstated, with ample resources available for those interested in its implementation.

Setting up JMX

JMX isn't enabled by default, to enable it we need the following steps:

1. Modify the JVM Startup Parameters: To enable JMX on a Java application, you must adjust the Java Virtual Machine (JVM) startup parameters. This involves adding specific flags to your application's startup command. The essential flags for enabling JMX are:

   • -Dcom.sun.management.jmxremote: This flag activates the JMX remote management and monitoring.

   • -Dcom.sun.management.jmxremote.port=<PORT>: Replace <PORT> with a specific port number where the JMX remote connection will listen.

   • -Dcom.sun.management.jmxremote.ssl=false: This flag disables SSL for JMX connections. For development environments, SSL might be disabled for simplicity, but for production environments, consider enabling SSL for security.

   • -Dcom.sun.management.jmxremote.authenticate=false: This flag disables authentication. Similar to SSL, authentication may be disabled in development but should be enabled in production to ensure secure access.

2. Restart Your Application: With the JVM parameters set, restart your application. This will apply the new startup parameters, activating JMX.

3. Verify JMX Connectivity: After restarting your application, you can verify that JMX is enabled by connecting to it using a JMX client such as JConsole, VisualVM, or a custom management application. Use the port number specified in the startup parameters to establish the connection.

JMX Security Considerations

While enabling JMX provides powerful management capabilities, it's crucial to consider security implications, especially when JMX is exposed over a network. When deploying applications in production, always enable SSL and authentication to protect against unauthorized access. Additionally, consider firewall rules and network policies to restrict JMX access to trusted clients.

Understanding MBeans

Central to JMX are Management Beans (MBeans), which serve as the control points within an application. These beans enable developers to publish specific functionalities for runtime monitoring and configuration. The ability to export application metrics to dashboards through MBeans is particularly valuable, facilitating real-time decision-making based on accurate, up-to-date information. Furthermore, operations such as user management can be exposed through MBeans, enhancing administrative capabilities.

Spring and Management Beans

Spring Framework's Actuator module exemplifies the integration of management capabilities within development, offering extensive metrics and operational details. This integration propels applications to "production-ready" status, allowing developers to monitor and manage applications with unprecedented depth and efficiency.

Tooling for JMX Management

While JMX can be accessed through various web interfaces and administrative tools, command-line tooling offers a direct, efficient method for interacting with JMX-enabled applications on production servers. Tools like JMXTerm complement visual tools by providing a streamlined interface for rapid insights, especially in environments unfamiliar to the developer.

Getting Started with JMXTerm

JMXTerm is a powerful utility for managing JMX without the need for graphical visualization, ideal for quick diagnostics or high-level server insights. After enabling JMX on the JVM and setting up the necessary configurations, developers can connect to servers, explore different JMX domains, and manipulate MBeans directly from the command line.

We can accomplish all of the following via visual tools and sometimes using a web interface. Normally, that's the approach I use. However, as a learning tool I think JMXTerm is fantastic since it exposes things in a way that's consistent and verbose. If we can understand JMXTerm the GUI version would be a walk in the park...

We can launch JMXTerm using the command line, in my case I used the following command:

java -jar ~/Downloads/jmxterm-1.0.2-uber.jar --url localhost:30002

Once the connection is made we can issue commands to JMX and retrieve information about the JVM or the application e.g. I can list the domains which you can think of as similar to "packages" or "modules" a way to organize the various beans:

$>domains
#following domains are available
JMImplementation
com.sun.management
java.lang
java.nio
java.util.logging
javax.cache
jdk.management.jfr

I can select a specific domain and thus perform future operations within said domain:

$>domain java.util.logging
#domain is set to java.util.logging

Once inside the domain I can select a specific bean and perform operations on it. For this I need to first list the beans in the domain, in this case there's only the logging bean. I can then select that bean using the bean command:

$>beans
#domain = java.util.logging:
java.util.logging:type=Logging
$>bean java.util.logging:type=Logging
#bean is set to java.util.logging:type=Logging

I can perform many operations on beans, perhaps the most useful is the info command which lets me query a bean. Notice that a bean can have attributes, think of them like object fields and operations which you can think of as methods. There are also notifications which you can think of as events:

$>info
#mbean = java.util.logging:type=Logging
#class name = sun.management.ManagementFactoryHelper$PlatformLoggingImpl
# attributes
  %0   - LoggerNames ([Ljava.lang.String;, r)
  %1   - ObjectName (javax.management.ObjectName, r)
# operations
  %0   - java.lang.String getLoggerLevel(java.lang.String p0)
  %1   - java.lang.String getParentLoggerName(java.lang.String p0)
  %2   - void setLoggerLevel(java.lang.String p0,java.lang.String p1)
#there's no notifications

I can run operations and pass various commands e.g. I can get the logger level, set it and then check that the logger level was indeed updated:

$>run getLoggerLevel "org.apache.tomcat.websocket.WsWebSocketContainer"
#calling operation getLoggerLevel of mbean java.util.logging:type=Logging with params [org.apache.tomcat.websocket.WsWebSocketContainer]
#operation returns:
$>run setLoggerLevel org.apache.tomcat.websocket.WsWebSocketContainer INFO
#calling operation setLoggerLevel of mbean java.util.logging:type=Logging with params [org.apache.tomcat.websocket.WsWebSocketContainer, INFO]
#operation returns: 
null
$>run getLoggerLevel "org.apache.tomcat.websocket.WsWebSocketContainer"
#calling operation getLoggerLevel of mbean java.util.logging:type=Logging with params [org.apache.tomcat.websocket.WsWebSocketContainer]
#operation returns: 
INFO

This is just the tip of the iceberg. We can get many things such as spring settings, internal VM information, etc. In this example I can query VM information directly from the console:

$>domain com.sun.management
#domain is set to com.sun.management
$>beans
#domain = com.sun.management:
com.sun.management:type=DiagnosticCommand
com.sun.management:type=HotSpotDiagnostic
$>bean com.sun.management:type=HotSpotDiagnostic
#bean is set to com.sun.management:type=HotSpotDiagnostic
$>info
#mbean = com.sun.management:type=HotSpotDiagnostic
#class name = com.sun.management.internal.HotSpotDiagnostic
# attributes
  %0   - DiagnosticOptions ([Ljavax.management.openmbean.CompositeData;, r)
  %1   - ObjectName (javax.management.ObjectName, r)
# operations
  %0   - void dumpHeap(java.lang.String p0,boolean p1)
  %1   - javax.management.openmbean.CompositeData getVMOption(java.lang.String p0)
  %2   - void setVMOption(java.lang.String p0,java.lang.String p1)
#there's no notifications

Leveraging JMX in Debugging and Management

JMX stands out as a robust tool for wiring management consoles, allowing developers to expose critical settings and metrics for their projects. Beyond its conventional use, JMX can be leveraged as part of the debugging process, serving as a pseudo-interface for triggering debugging scenarios or observing debugging sessions within the management UI. This approach not only simplifies the management of server applications but also enhances the developer's ability to diagnose and resolve issues efficiently.

Exposing MBeans in Spring Boot

Up until now we discussed the process of working with beans that are a part of the JVM or Spring. But what about our own application logic?

We can expose our own applications internal state so we (and our SREs) can review these in production and staging. Instead of building a custom control panel or logging everything, we can just expose the data. If a flag is problematic we can change it in production, if you want to query a specific state it too can be exposed.

Spring Boot simplifies the management and monitoring of applications through its comprehensive support for JMX. By leveraging Spring's infrastructure, we can easily expose their application's beans as JMX Managed Beans (MBeans), making them accessible for monitoring and management via JMX clients.

Understanding Spring Boot JMX Support

Spring Boot automatically configures JMX for you and exposes any beans annotated with @ManagedResource as JMX MBeans. This feature, combined with Spring Boot’s Actuator, provides a rich set of management endpoints, covering various aspects of the application, from metrics to thread dumps.

Expose an MBean in Spring Boot

To expose a bean we need to take the following steps:

1. Define a Management Interface: Create an interface that defines the operations and attributes you wish to expose via JMX. This interface should be annotated with JMX annotations such as @ManagedOperation for methods and @ManagedAttribute for fields or getter/setter methods.

2. Implement the MBean: Implement the interface in a class that performs the actual logic for the operations and attributes defined. This class represents your MBean and can be a regular Spring-managed bean.

3. Annotate the Bean with@ManagedResource: Annotate your MBean implementation class with @ManagedResource to indicate that it should be exposed as an MBean. You can specify the object name for the MBean in this annotation, which is how it will be identified in JMX clients.

4. Enable JMX in Spring Boot: Ensure that JMX is enabled in your Spring Boot application. This is usually the default behavior, but you can explicitly enable it by setting spring.jmx.enabled=true in your application.properties or application.yml file.

5. Access the MBean via a JMX Client: Once your application is running, you can access the exposed MBean through any standard JMX client, such as JConsole, VisualVM, or a custom client. Connect to the Spring Boot application's JMX domain, and you'll find the MBean you exposed, ready for interaction.

Example: Exposing a Simple Configuration MBean

// Define a management interface
public interface ConfigurationMBean {
    @ManagedAttribute
    String getApplicationName();

    @ManagedOperation
    void updateApplicationName(String name);
}

// Implement the MBean
@Component
@ManagedResource(objectName = "com.example:type=Configuration")
public class Configuration implements ConfigurationMBean {
    private String applicationName = "MyApp";

    @Override
    public String getApplicationName() {
        return applicationName;
    }

    @Override
    public void updateApplicationName(String name) {
        this.applicationName = name;
    }
}

In this example, the Configuration class is annotated with @ManagedResource, making it available as an MBean with operations and attributes accessible via JMX clients.

Exposing MBeans in Spring Boot is a powerful feature that enhances the management and monitoring capabilities of applications. By following the steps outlined above, developers can provide external tools with dynamic access to application internals, offering a window into the runtime behavior and allowing for adjustments on the fly. This not only aids in debugging and performance tuning but also aligns with best practices for building manageable, robust applications.

Final Word

Advanced management tools, particularly JMX and its integration with frameworks like Spring, offer developers powerful capabilities for application monitoring, configuration, and debugging. By understanding and utilizing these tools, developers can achieve a deeper level of control over their applications, enhancing both performance and reliability. Whether through graphical interfaces or command-line utilities like JMXTerm, the dynamic manipulation and monitoring of applications in runtime environments open new avenues for effective software development and management. As the bridge between development and operations continues to narrow, mastering these advanced tools becomes essential for any developer looking to excel in today's fast-paced technological landscape.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

The Essence of Debugging with Git

Before we tap into the advanced aspects of git bisect, it's essential to understand its foundational premise. Git is known for tracking changes and managing code history, but the git bisect tool is a hidden gem for regression detection. Regressions are distinct from generic bugs, they signify a backward step in functionality—where something that once worked flawlessly now fails. Pinpointing the exact change causing a regression can be akin to finding a needle in a haystack, particularly in extensive codebases with long commit histories.

Traditionally, developers would employ a manual, binary search strategy—checking out different versions, testing them, and narrowing down the search scope. This method, while effective, is painstakingly slow and error-prone. Git bisect automates this search, transforming what used to be a marathon into a swift sprint.

Setting the Stage for Debugging

Imagine you're working on a project, and recent reports indicate a newly introduced bug affecting the functionality of a feature that previously worked flawlessly. You suspect a regression but are unsure which commit introduced the issue among the hundreds made since the last stable version.

Initiating Bisect Mode

To start, you'll enter bisect mode in your terminal within the project's Git repository:

git bisect start

This command signals Git to prepare for the bisect process.

Marking the Known Good Revision

Next, you identify a commit where the feature functioned correctly, often a commit tagged with a release number or dated before the issue was reported. Mark this commit as "good":

git bisect good a1b2c3d

Here, a1b2c3d represents the hash of the known good commit.

Marking the Known Bad Revision

Similarly, you mark the current version or a specific commit where the bug is present as "bad":

git bisect bad z9y8x7w

z9y8x7w is the hash of the bad commit, typically the latest commit in the repository where the issue is observed.

Bisecting to Find the Culprit

Upon marking the good and bad commits, Git automatically jumps to a commit roughly in the middle of the two and waits for you to test this revision. After testing (manually or with a script), you inform Git of the result:

• If the issue is present: git bisect bad

• If the issue is not present: git bisect good

Git then continues to narrow down the range, selecting a new commit to test based on your feedback.

Expected Output

After several iterations, Git will isolate the problematic commit, displaying a message similar to:

Bisecting: 0 revisions left to test after this (roughly 3 steps)
[abcdef1234567890] Commit message of the problematic commit

Reset and Analysis

Once the offending commit is identified, you conclude the bisect session to return your repository to its initial state:

git bisect reset

Notice that bisect isn't linear. Bisect doesn't scan through the revisions in a sequential manner. Based on the good and bad markers, Git automatically selects a commit approximately in the middle of the range for testing (e.g., commit #6 in the following diagram). This is where the non-linear, binary search pattern starts, as Git divides the search space in half instead of examining each commit sequentially. This means fewer revisions get scanned and the process is faster.

Advanced Usage and Tips

The magic of git bisect lies in its ability to automate the binary search algorithm within your repository, systematically halving the search space until the rogue commit is identified.

Git bisect offers a powerful avenue for debugging, especially for identifying regressions in a complex codebase. To elevate your use of this tool, consider delving into more advanced techniques and strategies. These tips not only enhance your debugging efficiency but also provide practical solutions to common challenges encountered during the bisecting process.

Script Automation for Precision and Efficiency

Automating the bisect process with a script is a game-changer, significantly reducing manual effort and minimizing the risk of human error. This script should ideally perform a quick test that directly targets the regression, returning an exit code based on the test's outcome.

Example: Imagine you're debugging a regression where a web application's login feature breaks. You could write a script that attempts to log in using a test account and checks if the login succeeds. The script might look something like this in a simplified form:

#!/bin/bash
# Attempt to log in and check for success
if curl -s http://yourapplication/login -d "username=test&password=test" | grep -q "Welcome"; then
  exit 0 # Login succeeded, mark this commit as good
else
  exit 1 # Login failed, mark this commit as bad
fi

By passing this script to git bisect run, Git automatically executes it at each step of the bisect process, effectively automating the regression hunt.

Handling Flaky Tests with Strategy

Flaky tests, which sometimes pass and sometimes fail under the same conditions, can complicate the bisecting process. To mitigate this, your automation script can include logic to rerun tests a certain number of times or to apply more sophisticated checks to differentiate between a true regression and a flaky failure.

Example: Suppose you have a test that's known to be flaky. You could adjust your script to run the test multiple times, considering the commit "bad" only if the test fails consistently:

#!/bin/bash
# Run the flaky test three times
success_count=0
for i in {1..3}; do
  if ./run_flaky_test.sh; then
    ((success_count++))
  fi
done

# If the test succeeds twice or more, consider it a pass
if [ "$success_count" -ge 2 ]; then
  exit 0
else
  exit 1
fi

This approach reduces the chances that a flaky test will lead to incorrect bisect results.

Skipping Commits with Care

Sometimes, you'll encounter commits that cannot be tested due to reasons like broken builds or incomplete features. git bisect skip is invaluable here, allowing you to bypass these commits. However, use this command judiciously to ensure it doesn't obscure the true source of the regression.

Example: If you know that commits related to database migrations temporarily break the application, you can skip testing those commits. During the bisect session, when Git lands on a commit you wish to skip, you would manually issue:

git bisect skip

This tells Git to exclude the current commit from the search and adjust its calculations accordingly. It's essential to only skip commits when absolutely necessary, as skipping too many can interfere with the accuracy of the bisect process.

These advanced strategies enhance the utility of git bisect in your debugging toolkit. By automating the regression testing process, handling flaky tests intelligently, and knowing when to skip untestable commits, you can make the most out of git bisect for efficient and accurate debugging. Remember, the goal is not just to find the commit where the regression was introduced but to do so in the most time-efficient manner possible. With these tips and examples, you're well-equipped to tackle even the most elusive regressions in your projects.

Unraveling a Regression Mystery

In the past we got to use git bisect, when working on a large-scale web application. After a routine update, users began reporting a critical feature failure: the application's payment gateway stopped processing transactions correctly, leading to a significant business impact.

We knew the feature worked in the last release but had no idea which of the hundreds of recent commits introduced the bug. Manually testing each commit was out of the question due to time constraints and the complexity of the setup required for each test.

Enter git bisect. The team started by identifying a "good" commit where the payment gateway functioned correctly and a "bad" commit where the issue was observed. We then crafted a simple test script that would simulate a transaction and check if it succeeded.

By running git bisect start, followed by marking the known good and bad commits, and executing the script with git bisect run, we set off on an automated process that identified the faulty commit. Git efficiently navigated through the commits, automatically running the test script on each step. In a matter of minutes, git bisect pinpointed the culprit: a seemingly innocuous change to the transaction logging mechanism that inadvertently broke the payment processing logic.

Armed with this knowledge we reverted the problematic change, restoring the payment gateway's functionality and averting further business disruption. This experience not only resolved the immediate issue but also transformed our approach to debugging, making git bisect a go-to tool in our arsenal.

Final Word

The story of the payment gateway regression is just one example of how git bisect can be a lifesaver in the complex world of software development. By automating the tedious process of regression hunting, git bisect not only saves precious time but also brings a high degree of precision to the debugging process.

As developers continue to navigate the challenges of maintaining and improving complex codebases, tools like git bisect underscore the importance of leveraging technology to work smarter, not harder. Whether you're dealing with a mysterious regression or simply want to refine your debugging strategies, git bisect offers a powerful, yet underappreciated, solution to swiftly and accurately identify the source of regressions. Remember, the next time you're faced with a regression, git bisect might just be the debugging partner you need to uncover the truth hidden within your commit history.

"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements." -- Brian Kernighan.

Cutting a patient open and using print for debugging used to be the best ways to diagnose problems. If you still advocate either one of those as the superior approach to troubleshooting then you're either facing a very niche problem or need to update your knowledge. This is a frequent occurrence, e.g this recent tweet:

This specific tweet got to the HN front page and people chimed in with that usual repetitive nonsense. No, it’s not the best way for the vast majority of developers. It should be discouraged just as surgery should be avoided when possible.

Fixating on print debugging is a form of a mental block, debugging isn’t just stepping over code. It requires a completely new way of thinking about issue resolution. A way that is far superior to merely printing a few lines.

Before I continue, my bias is obvious. I wrote a book about debugging and I blog about it a lot. This is a pet peeve of mine.

I want to start with the exception to the rule though, when do we need to print something...

Logging is NOT Print Debugging!

One of the most important debugging tools in our arsenal is a logger, but it is not the same as print debugging in any way:

A log is something we add with forethought, we want to keep the log for future bugs and might even want to expose it to the users. We can control its verbosity often at the module level and can usually disable it entirely. It’s permanent in code and usually writes to a permanent file we can review at our leisure.

Print debugging is code we add to locate a temporary problem. If such a problem has the potential of recurring then a log would typically make more sense in the long run. This is true for almost every type of system, we see developers adding print statements and removing them constantly instead of creating a simple log to track frequent problems.

There are special cases where print debuggings makes some sense, in mission critical embedded systems a log might be impractical in terms of device constraints. Debuggers are awful in those environments and print debugging is a simple hack. Debugging system level tools like a kernel, compiler, debugger,or JIT can be difficult with a debugger. Logging might not make sense in all of these cases e.g. I don’t want my JIT to print every bytecode it’s processing and the metadata involved.

Those are the exceptions, not the rules. Very few of us write such tools. I do, and even then it’s a fraction of my work. E.g. When working at Lightrun I was working on a production debugger. Debugging the agent code that’s connected to the executable was one of the hardest things to do. A mix of C++ and JVM code that’s connected to a completely separate binary... Print debugging of that portion was simpler, and even then we tried to aim towards logging. But the visual aspects of the debugger within the server backend and the IDE were perfect targets for the debugger.

Why Debug?

There are three reasons to use a debugger instead of printouts or even logs:

• Features - modern debuggers can provide spectacular capabilities that are unfamiliar to many developers. Sadly, there are very few debugging courses in academia since it’s a subject that’s hard to test.

• Low overhead - in the past running with the debugger meant slow execution and a lot of overhead. This is no longer true. Many of us use the debug action when launching an application instead of running and there’s no noticeable overhead for most applications. When there is overhead, some debuggers provide means to improve performance by disabling some features.

• Library code - a debugger can step into a library or framework and track the bug there. Doing this with print debugging will require compiling code that you might not want to deal with.

I dug into the features I mentioned in my book and series on debugging (linked above) but let’s pick a few fantastic capabilities of the debugger that I wrote about in the past.

For the sake of positive dialog, here are some of my top features of modern debuggers.

Tracepoints

Whenever someone opens the print debugging discussion all I hear is “I don’t know about tracepoints”. They aren’t a new feature in debuggers, yet so few are aware of them. A tracepoint is a breakpoint that doesn’t stop, it just keeps running. Instead of stopping you can do other things at that point, such as print to the console. This is similar to print debugging only it doesn’t suffer from many of the drawbacks: no runtime overhead, no accidental commit to the code base, no need to restart the application when changing it etc.

Grouping and Naming

The previous video/post included a discussion of grouping and naming. This lets us group tracepoints together, disable them as a group etc. This might seem like a minor feature until you start thinking about the process of print debugging. We slowly go through the code adding a print and restarting. Then suddenly we need to go back, or if a call comes in and we need to debug something else...

When we package the tracepoints and breakpoints into a group we can set aside a debugging session much like we set aside a branch in version control. It makes it much easier to preserve our train of thought and jump right back to the applicable lines of code.

Object Marking

When asked about my favorite debugging feature I’m always conflicted, Object Marking is one of my top two features... It seems like a simple thing, we can mark an object and it gets saved with a specific name.

However, this is a powerful and important feature. I used to write down the pointers to objects or memory areas while debugging. This is valuable as sometimes an area of memory would look the same but would have a different address or it might be hard to track objects with everything going on. Object Marking allows us to save a global reference to an object, use it in conditional breakpoints or for visual comparison.

Renderers

My other favorite feature is the renderer, it lets us define how elements look in the debugger watch area. Imagine you have a sophisticated object hierarchy but rarely need that information... A renderer lets you customize the way IntelliJ/IDEA presents the object to you.

Tracking New Instances

One of the often overlooked capabilities of the debugger is memory tracking. A Java debugger can show you a searchable set of all object instances in heap, that is a fantastic capability that can expose unintuitive behavior But it can go further, it can track new allocations of an object and provide you with the stack to the applicable object allocation.

Tip of the Iceberg

I wrote a lot about debugging, there’s no point in repeating all of that in this post. If you’re a person who feels more comfortable using print debugging then ask yourself this: why?

Don’t hide behind an out of date Brian Kernighan quote. Things change. Are you working in one of the edge cases where print debugging is the only option?

Are you treating logging as print debugging or vice versa?

Or is it just that print debugging was how your team always worked and it stuck in place. If it’s one of those then it might be time to re-evaluate the current state of debuggers.

gdocweb has its first few users and one of the big complaints is about the tediousness of going through that wizard every time you just want to test a change to a document. I don’t want gdocweb working in the background reading my documents and publishing, that would be an invasion of privacy. But I would like it to work instantly.

With that we now have a link to regenerate the site quickly. Not with a single click, you still need to go through a Google login for security. Once that’s done you will reach the final stage of the wizard directly and the website will be updated. You can do that by visiting https://gdocweb.com/regenerateSite.

RSS (Really Simple Syndication) and Sitemap

Many of us go through life without knowing the RSS exists, it’s a workhorse that powers a great deal of functionality on the internet. Yet, we remain oblivious to it. RSS lets a website broadcast about the changes it went through, e.g. this blog now features an RSS feed that can notify you about every new post to the site.

Typically one would read an RSS feed using a dedicated application (e.g. Feedly), but browsers also have some basic support for RSS. The biggest benefit of RSS is in syndication, it means that other sites can publish a “feed” from this site detailing the latest bit of news. It’s great for search engine optimization and a wonderful feature for users of your website. You can see the feed.xml RSS file here.

Sitemaps are even more important. They let search engines know about the pages you have in the website and the dates in which they were last updated. This helps search engines keep track of everything and makes your site easier to find. The sitemap for this site can be found here. Normally you wouldn’t care about it, but search engines care about it...

Both files require a full URL to the generated pages in order to function. Unfortunately, this isn’t trivial. The gdocweb blog can be reached both on: https://shai-almog.github.io/GdocwebBlog/ and on https://blog.gdocweb.com/. The latter is the correct link and the former correctly redirects to it, however if we have links to the former this will reduce the search engine ranking. We need to link to the correct blog URL but gdocweb can’t guess it from the project name.

That’s why all of this good stuff will only work if you set the value of the “Base URL” entry in the GitHub repository selection stage of the wizard. Once that is set as shown in the following image, this will all work as expected.

As a bonus, setting this value will also set the canonical URL for each page. This is an important attribute of an HTML file that helps search engines find your website.

Target Directory and Automatic Merge

In the previous image we could see two additional features that are also quite important but mostly geared towards the technical crowd. The first is the target directory. By default, gdocweb generates everything into the GitHub Projects root directory. This is great for a site, however if you’re building documentation for a pre-existing project then generating the project to a docs directory might be a better approach. The “Target Directory” option is a great tool for developers building documentation and websites for their projects.

gdocweb generates a pull request for the project and merges that pull request for you automatically. This default behavior might not be right for all projects and also includes a risk. You might want an additional review for changes. In that case you can uncheck the “Merge Automatically” flag and disable the default behavior. This means a run of the gdocweb wizard will result with a new pull request for you to merge manually. For me that has been a valuable debugging tool as I could experiment with changes to the blog without merging them in.

You can see the document that generated this post here.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Understanding strace and its Origins

A Look Back: strace and dtrace

We discussed dtrace the lasttime around. However, dtrace's availability is limited, particularly on Linux systems where most server and container debugging takes place. This is where strace comes into the picture, offering a simpler yet effective alternative.

Originating from Sun Microsystems

strace, like dtrace, traces its roots back to Sun Microsystems, emerging in the 90s (a decade before dtrace). This isn't surprising given the impressive array of technologies that originated from Sun. However, strace differentiates itself by its straightforwardness in both usage and capabilities. Unlike DTrace, which demands deep operating system support and thus remained absent as an official feature in common Linux distributions, strace thrives in the Linux environment. Its simplicity and ease of implementation make it a popular choice for Linux users, offering a distinct approach to system diagnostics.

Technical Functioning of strace

The Role of ptrace in strace

The cornerstone of strace's functionality is the ptrace kernel feature. ptrace, pre-existing in Linux, spares users from the need to add additional kernel code or modules, a requirement often associated with DTrace. This fundamental difference not only simplifies the use of strace but also broadens its accessibility.

Comparing with DTrace

While DTrace offers a more in-depth analysis through deeper kernel support, strace operates on a more surface level. This simplicity, however, does not undermine its effectiveness. strace works essentially by logging every kernel call made by a process, providing verbose but incredibly detailed insights into the system's operation. This method allows users to trace the inner workings of a process, understanding each interaction with the kernel.

Practical Usage and Advantages

Ease of Use and Accessibility

One of the most appealing aspects of strace is its user-friendly nature. It doesn't require special privileges or complex setup procedures. This ease of use is particularly beneficial for developers and system administrators who need to quickly diagnose and address issues in a Linux environment. Unlike DTrace, strace is readily available and doesn’t demand advanced configurations or permissions.

Favored in Linux Environments

strace's popularity in Linux circles is not only due to its accessibility but also its practicality. Being able to run without special privileges makes it a go-to tool for diagnosing various system-related issues. However, it's important to note that strace should be used cautiously in production environments. Its extensive logging can create a significant performance overhead, potentially impacting the efficiency of a live system. This is why strace is generally recommended for use in development or isolated testing environments rather than in production.

strace in Action: A Closer Look at System Calls

Basic Usage and Output Analysis

Using strace is straightforward: you simply pass the command line to it.

strace java -classpath . PrimeMain

This simplicity belies its power, as the output offers a wealth of information. Each line in the strace output corresponds to a system call made by the process as you can see below:

execve("/home/ec2-user/jdk1.8.0_45/bin/java", ["java", "-classpath.", "PrimeMain"], 0x7fffd689ec20 /* 23 vars */) = 0
brk(NULL)                               = 0xb85000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0294272000
readlink("/proc/self/exe", "/home/ec2-user/jdk1.8.0_45/bin/j"..., 4096) = 35
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64", 0x7fff37af09a0) = -1 ENOENT (No such file or directory)
open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls", 0x7fff37af09a0) = -1 ENOENT (No such file or directory)

By analyzing these calls, users can gain insights into the intricate operations of their applications. For instance, if a Java process attempts to load a library and fails, strace can reveal the underlying system call and its exit code, providing clues about potential issues like missing files or directories. E.g. in this line:

open("/home/ec2-user/jdk1.8.0_45/bin/../lib/amd64/jli/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Java tries to load the pthread library from the tls directory using a system call open to load the file. The exit code of the system call is -1, which means that the file isn't there. Under normal circumstances, we should get back a file descriptor value from this API (positive non-zero integer). Looking in the directory, it seems the tls directory is missing. I'm guessing that this is because of a missing JCE (Java Cryptography Extensions) installation. This is probably OK but might have been interesting in some cases.

Interpreting System Calls for Debugging

The output of strace, while verbose, is a goldmine for troubleshooting. For example, a negative exit code in a system call indicates an error, such as a missing file, which could be crucial for diagnosing issues in an application. This level of detail, although overwhelming at times, is invaluable for understanding the interactions between your application and the Linux system.

Advanced Features and Tips

Filtering System Calls for Efficiency

A common challenge with strace is managing its voluminous output. Fortunately, strace offers options to filter system calls, significantly enhancing its usability. By using the -e argument, you can instruct strace to log only specific types of system calls, such as open or connect e.g.:

strace -e open java -classpath . PrimeMain

This selective logging not only makes the output more manageable but also allows for focused troubleshooting, speeding up the debugging process.

Exploring a Variety of System Calls

strace's utility extends beyond just tracking file access or network interactions. It can be used to monitor a range of system calls, offering insights into various aspects of application behavior. By understanding and utilizing different system calls, users can gain a comprehensive view of their application's interaction with the operating system, leading to more effective debugging and optimization.

strace and Java: A Special Case

strace with the JVM

While strace predates Java and operates at a low level with no specific awareness of the Java Virtual Machine (JVM), it remains highly effective for debugging Java applications. The JVM, like most platforms, relies on system calls for its operations, which strace can monitor and report. However, certain aspects of the JVM's behavior may be less visible to strace due to its unique approach to problem-solving.

Allocations and Threading in Java

For instance, Java's memory management differs significantly from standard system tools. While typical applications use malloc, which directly maps to kernel allocation logic, Java manages its own memory. This approach, aimed at efficiency and streamlined garbage collection, means that some memory allocation activities are obscured from strace's view.

Similarly, Java threading is currently well-represented in strace output, but this is changing with Java 21 and Project Loom. Java 21 added support for Virtual Threads which are only partially visible to the operating system hence 1,000 threads can seem like 16 threads. These changes could affect the clarity of strace outputs in complex, heavily threaded Java applications.

Final Word

strace stands out as an exceptionally versatile and powerful tool in the Linux debugging arsenal. Its ability to provide detailed insights into system calls makes it invaluable for diagnosing and understanding the inner workings of applications. Despite its simplicity, strace is capable of handling complex debugging scenarios, especially when used with its advanced filtering options.

For developers and system administrators working in Linux environments, strace is more than just a diagnostic tool; it's a lens through which the intricate interactions between applications and the operating system can be viewed and understood. As technologies evolve, tools like strace adapt, continuing to offer relevant and critical insights into system behaviors.

Whether you are troubleshooting a stubborn issue or simply curious about how your applications interact with the Linux kernel, strace is a tool that you will likely find yourself returning to time and again.

Styling a Document isn’t as Trivial as you Might Assume

When using Google Docs it’s often tempting to reach out to the font or color toolbar to design your document as you see fit. This would work great for simple sites but as gdocweb evolves you might find it produces a result that isn’t as attractive as you might want. The reason for that is due to two important features we just added to gdocweb: custom colors and dark mode.

Dark Mode Support

We recently updated gdocweb with a new theme: Adaptive.

The Adaptive theme is identical to the default “Basic” theme but when it’s running on a device set to dark mode it will render the website with a dark version of the theme. As you can see below, this blog automatically adapts to light or dark mode. However, the transition wasn’t seamless.

Title Styling

The initial design of the blog was based on a template from Google Docs. In this template “Introducing gdocweb” was defined as a “Heading 1” style, but it was later customized to appear a bit differently from the actual “Heading 1” definition.

Headings are an important part of the web, in general your page should have only one “Heading 1” and multiple “Heading 2” elements (and possibly 3, 4, 5 etc.). If you break these rules you might suffer a search engine penalty. But it goes deeper than that, once you customize the style of the heading we can’t override it. That results in dark blue text on black background. Not very readable for the most important line within a post...

The solution is simple and it includes two parts. The first is to use styles in the document by placing the cursor on the heading and invoking “Update ‘Heading 1’ to match (or the appropriate style you want to customize). We then need to select “Apply ‘Heading 1’” to re-apply it to the current line. Besides the obvious advantages with dark mode, this also helps keep the document consistent. E.g. if we want to change the font or color for all “Heading 2” entries we can do that in one place and they will all update.

But the real advantage is in gdocweb. We can now customize specific colors within the theme. This is useful for dark mode but also useful for applying brand colors on top of the theme. We can pick replacement default colors for background and foreground of various elements and make the replacements apply only for light or dark mode. This is all a part of the themes section as seen in the image below.

Future Improvements

Ideally the themes should include the right colors out of the box but the ability to customize these colors is a crucial one. Let us know in the comments if you need deeper styling and what your technical level is. One of our planned features is a style override that will let you inject a custom style file to the page. However, this would require technical understanding and control of CSS, I’m not sure if this is something that would fit with our main demographic.

You can see the document that generated this post here.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

DTrace Overview

DTrace was first introduced by Sun Microsystems in 2004, DTrace quickly garnered attention for its groundbreaking approach to dynamic system tracing. Originally developed for Solaris, it has since been ported to various platforms, including MacOS, Windows, and Linux. DTrace stands out as a dynamic tracing framework that enables deep inspection of live systems – from operating systems to running applications. Its capacity to provide real-time insights into system and application behavior without significant performance degradation marks it as a revolutionary tool in the domain of system diagnostics and debugging.

Understanding DTrace’s Capabilities

DTrace, short for Dynamic Tracing, is a comprehensive toolkit for real-time system monitoring and debugging, offering an array of capabilities that span across different levels of system operation. Its versatility lies in its ability to provide insights into both high-level system performance and detailed process-level activities.

System Monitoring and Analysis

At its core, DTrace excels in monitoring various system-level operations. It can trace system calls, file system activities, and network operations. This enables developers and system administrators to observe the interactions between the operating system and the applications running on it. For instance, DTrace can identify which files a process accesses, monitor network requests, and even trace system calls to provide a detailed view of what's happening within the system.

Process and Performance Analysis

Beyond system-level monitoring, DTrace is particularly adept at dissecting individual processes. It can provide detailed information about process execution, including CPU and memory usage, helping to pinpoint performance bottlenecks or memory leaks. This granular level of detail is invaluable for performance tuning and debugging complex software issues.

Customizability and Flexibility

One of the most powerful aspects of DTrace is its customizability. With a scripting language based on C syntax, DTrace allows the creation of customized scripts to probe specific aspects of system behavior. This flexibility means that it can be adapted to a wide range of debugging scenarios, making it a versatile tool in a developer’s arsenal.

Real-World Applications

In practical terms, DTrace can be used to diagnose elusive performance issues, track down resource leaks, or understand complex interactions between different system components. For example, it can be used to determine the cause of a slow file operation, analyze the reasons behind a process crash, or understand the system impact of a new software deployment.

Performance and Compatibility of DTrace

A standout feature of DTrace is its ability to operate with remarkable efficiency. Despite its deep system integration, DTrace is designed to have minimal impact on overall system performance. This efficiency makes it a feasible tool for use in live production environments, where maintaining system stability and performance is crucial. Its non-intrusive nature allows developers and system administrators to conduct thorough debugging and performance analysis without the worry of significantly slowing down or disrupting the normal operation of the system.

Cross-Platform Compatibility

Originally developed for Solaris, DTrace has evolved into a cross-platform tool, with adaptations available for MacOS, Windows, and various Linux distributions. Each platform presents its own set of features and limitations. For instance, while DTrace is a native component in Solaris and MacOS, its implementation in Linux often requires a specialized build due to kernel support and licensing considerations.

Compatibility Challenges on MacOS

On MacOS, DTrace's functionality intersects with System Integrity Protection (SIP), a security feature designed to prevent potentially harmful actions. To utilize DTrace effectively, users may need to disable SIP, which should be done with caution. This process involves booting into recovery mode and executing specific commands, a step that highlights the need for a careful approach when working with such powerful system-level tools.

We can disable SIP using the command:

csrutil disable

We can optionally use a more refined approach of enabling SIP without dtrace using the following command:

csrutil enable --without dtrace

Be extra careful when issuing these commands and when working on machines where dtrace is enabled. Back up your data properly!

Customizability and Flexibility of DTrace

A key feature that sets DTrace apart in the realm of system monitoring tools is its highly customizable nature. DTrace employs a scripting language that bears similarity to C syntax, offering users the ability to craft detailed and specific diagnostic scripts. This scripting capability allows for the creation of custom probes that can be fine-tuned to target particular aspects of system behavior, providing precise and relevant data.

Adaptability to Various Scenarios

The flexibility of DTrace's scripting language means it can adapt to a multitude of debugging scenarios. Whether it's tracking down memory leaks, analyzing CPU usage, or monitoring I/O operations, DTrace can be configured to provide insights tailored to the specific needs of the task. This adaptability makes it an invaluable tool for both developers and system administrators who require a dynamic approach to problem-solving.

Examples of Customizable Probes

Users can define probes to monitor specific system events, track the behavior of certain processes, or gather data on system resource usage. This level of customization ensures that DTrace can be an effective tool in a variety of contexts, from routine maintenance to complex troubleshooting tasks. Following in a simple hello world dtrace probe:

sudo dtrace -qn 'syscall::write:entry, syscall::sendto:entry /pid == $target/ { printf("(%d) %s %s", pid, probefunc, copyinstr(arg1)); }' -p 9999

The kernel is instrumented with hooks that match various callbacks. dtrace connects to these hooks and can perform interesting tasks when these hooks are triggered. They have a naming convention, specially: provider:module:function:name. In this case the provider is a system call in both cases. We have no module so we can leave that part blank between the colon (:) symbols. We grab a write operation and sendto entries. When an application will write or tries to send a packet, the following code event will trigger.

These things happen frequently which is why we restrict the process ID to the specific target with pid == $target. This means the code will only trigger for the PID passed to us in the command line. The rest of the code should be simple for anyone with basic C experience, it's a printf that would list the processes and the data passed.

Real-World Applications of DTrace

DTrace's diverse capabilities extend far beyond theoretical use, playing a pivotal role in resolving real-world system complexities. Its ability to provide deep insights into system operations makes it an indispensable tool in a variety of practical applications.

To get a sense of how dtrace can be used we can use the man -k dtrace command whose output on my mac is below:

bitesize.d(1m)           - analyse disk I/O size by process. Uses DTrace
cpuwalk.d(1m)            - Measure which CPUs a process runs on. Uses DTrace
creatbyproc.d(1m)        - snoop creat()s by process name. Uses DTrace
dappprof(1m)             - profile user and lib function usage. Uses DTrace
dapptrace(1m)            - trace user and library function usage. Uses DTrace
dispqlen.d(1m)           - dispatcher queue length by CPU. Uses DTrace
dtrace(1)                - dynamic tracing compiler and tracing utility
dtruss(1m)               - process syscall details. Uses DTrace
errinfo(1m)              - print errno for syscall fails. Uses DTrace
execsnoop(1m)            - snoop new process execution. Uses DTrace
fddist(1m)               - file descriptor usage distributions. Uses DTrace
filebyproc.d(1m)         - snoop opens by process name. Uses DTrace
hotspot.d(1m)            - print disk event by location. Uses DTrace
iofile.d(1m)             - I/O wait time by file and process. Uses DTrace
iofileb.d(1m)            - I/O bytes by file and process. Uses DTrace
iopattern(1m)            - print disk I/O pattern. Uses DTrace
iopending(1m)            - plot number of pending disk events. Uses DTrace
iosnoop(1m)              - snoop I/O events as they occur. Uses DTrace
iotop(1m)                - display top disk I/O events by process. Uses DTrace
kill.d(1m)               - snoop process signals as they occur. Uses DTrace
lastwords(1m)            - print syscalls before exit. Uses DTrace
loads.d(1m)              - print load averages. Uses DTrace
newproc.d(1m)            - snoop new processes. Uses DTrace
opensnoop(1m)            - snoop file opens as they occur. Uses DTrace
pathopens.d(1m)          - full pathnames opened ok count. Uses DTrace
perldtrace(1)            - Perl's support for DTrace
pidpersec.d(1m)          - print new PIDs per sec. Uses DTrace
plockstat(1)             - front-end to DTrace to print statistics about POSIX mutexes and read/write locks
priclass.d(1m)           - priority distribution by scheduling class. Uses DTrace
pridist.d(1m)            - process priority distribution. Uses DTrace
procsystime(1m)          - analyse system call times. Uses DTrace
rwbypid.d(1m)            - read/write calls by PID. Uses DTrace
rwbytype.d(1m)           - read/write bytes by vnode type. Uses DTrace
rwsnoop(1m)              - snoop read/write events. Uses DTrace
sampleproc(1m)           - sample processes on the CPUs. Uses DTrace
seeksize.d(1m)           - print disk event seek report. Uses DTrace
setuids.d(1m)            - snoop setuid calls as they occur. Uses DTrace
sigdist.d(1m)            - signal distribution by process. Uses DTrace
syscallbypid.d(1m)       - syscalls by process ID. Uses DTrace
syscallbyproc.d(1m)      - syscalls by process name. Uses DTrace
syscallbysysc.d(1m)      - syscalls by syscall. Uses DTrace
topsyscall(1m)           - top syscalls by syscall name. Uses DTrace
topsysproc(1m)           - top syscalls by process name. Uses DTrace
Tcl_CommandTraceInfo(3tcl), Tcl_TraceCommand(3tcl), Tcl_UntraceCommand(3tcl) - monitor renames and deletes of a command
bitesize.d(1m)           - analyse disk I/O size by process. Uses DTrace
cpuwalk.d(1m)            - Measure which CPUs a process runs on. Uses DTrace
creatbyproc.d(1m)        - snoop creat()s by process name. Uses DTrace
dappprof(1m)             - profile user and lib function usage. Uses DTrace
dapptrace(1m)            - trace user and library function usage. Uses DTrace
dispqlen.d(1m)           - dispatcher queue length by CPU. Uses DTrace
dtrace(1)                - dynamic tracing compiler and tracing utility
dtruss(1m)               - process syscall details. Uses DTrace
errinfo(1m)              - print errno for syscall fails. Uses DTrace
execsnoop(1m)            - snoop new process execution. Uses DTrace
fddist(1m)               - file descriptor usage distributions. Uses DTrace
filebyproc.d(1m)         - snoop opens by process name. Uses DTrace
hotspot.d(1m)            - print disk event by location. Uses DTrace
iofile.d(1m)             - I/O wait time by file and process. Uses DTrace
iofileb.d(1m)            - I/O bytes by file and process. Uses DTrace
iopattern(1m)            - print disk I/O pattern. Uses DTrace
iopending(1m)            - plot number of pending disk events. Uses DTrace
iosnoop(1m)              - snoop I/O events as they occur. Uses DTrace
iotop(1m)                - display top disk I/O events by process. Uses DTrace
kill.d(1m)               - snoop process signals as they occur. Uses DTrace
lastwords(1m)            - print syscalls before exit. Uses DTrace
loads.d(1m)              - print load averages. Uses DTrace
newproc.d(1m)            - snoop new processes. Uses DTrace
opensnoop(1m)            - snoop file opens as they occur. Uses DTrace
pathopens.d(1m)          - full pathnames opened ok count. Uses DTrace
perldtrace(1)            - Perl's support for DTrace
pidpersec.d(1m)          - print new PIDs per sec. Uses DTrace
plockstat(1)             - front-end to DTrace to print statistics about POSIX mutexes and read/write locks
priclass.d(1m)           - priority distribution by scheduling class. Uses DTrace
pridist.d(1m)            - process priority distribution. Uses DTrace
procsystime(1m)          - analyse system call times. Uses DTrace
rwbypid.d(1m)            - read/write calls by PID. Uses DTrace
rwbytype.d(1m)           - read/write bytes by vnode type. Uses DTrace
rwsnoop(1m)              - snoop read/write events. Uses DTrace
sampleproc(1m)           - sample processes on the CPUs. Uses DTrace
seeksize.d(1m)           - print disk event seek report. Uses DTrace
setuids.d(1m)            - snoop setuid calls as they occur. Uses DTrace
sigdist.d(1m)            - signal distribution by process. Uses DTrace
syscallbypid.d(1m)       - syscalls by process ID. Uses DTrace
syscallbyproc.d(1m)      - syscalls by process name. Uses DTrace
syscallbysysc.d(1m)      - syscalls by syscall. Uses DTrace
topsyscall(1m)           - top syscalls by syscall name. Uses DTrace
topsysproc(1m)           - top syscalls by process name. Uses DTrace

There's a lot here, we don't need to read everything. The point is that when you run into a problem you can just search through this list and find a tool dedicated to debugging that problem.

Let’s say you're facing elevated disk write issues that are causing the performance of your application to degrade... But is it your app at fault or some other app?

rwbypid.d can help you with that, it can generate a list of processes and the number of calls they have for read/write based on the process id as seen in the following screenshot:

We can use this information to better understand IO issues in our code or even in 3rd party applications/libraries. iosnoop is another tool that helps us track IO operations but with more details:

In diagnosing elusive system issues, DTrace shines by enabling detailed observation of system calls, file operations, and network activities. For instance, it can be used to uncover the root cause of unexpected system behaviors or to trace the origin of security breaches, offering a level of detail that is often unattainable with other debugging tools.

Performance optimization is main area where DTrace demonstrates its strengths. It allows administrators and developers to pinpoint performance bottlenecks, whether they lie in application code, system calls, or hardware interactions. By providing real-time data on resource usage, DTrace helps in fine-tuning systems for optimal performance.

Final Words

In conclusion, DTrace stands as a powerful and versatile tool in the realm of system monitoring and debugging. We've explored its broad capabilities, from in-depth system analysis to individual process tracing, and its remarkable performance efficiency that allows for its use in live environments. Its cross-platform compatibility, coupled with the challenges and solutions specific to MacOS, highlights its widespread applicability. The customizability through scripting provides unmatched flexibility, adapting to a myriad of diagnostic needs. Real-world applications of DTrace in diagnosing system issues and optimizing performance underscore its practical value.

DTrace's comprehensive toolkit offers an unparalleled window into the inner workings of systems, making it an invaluable asset for system administrators and developers alike. Whether it's for routine troubleshooting or complex performance tuning, DTrace provides insights and solutions that are essential in the modern computing landscape.