It’s a new year and a great time to make changes for the better. In an effort to rid the world of needless network and application performance slowdowns, we turn to retransmission timeouts (RTOs). What are they and what can you do about them?

Read more on Performance Metric of the Month: Retransmission Timeouts (RTOs) and Application Performance Degradation…

Adrian Lane, CTO of Securosis, recently wrote an article titled Database Monitoring Best Practices: Using DAM Tools for SearchSecurity.com. The article plays on a security angle but provides a nice opportunity to discuss the parallels of monitoring database performance.

Read more on Application Performance Management (APM) + Database Activity Monitoring (DAM): It’s [Not] All About the Agents!…

In past blog posts, we’ve talked about performance metrics, such as TCP retransmission timeouts (RTOs) and database performance metrics, which help IT teams solve hard-to-pinpoint issues before they cause big problems. This month, we turn our attention to an especially tricky network performance metric: PAWS Dropped SYNs.

Read more on Performance Metric of the Month: PAWS Dropped SYNs and Network Address Translation…

CIFS Errors: The ExtraHop system provides storage performance details not contained in event logs.

September’s Performance Metric of the Month (PMM) focuses on CIFS errors for networked storage, but applies equally to other storage networking protocols such as NFS and iSCSI. Previous PMMs included TCP retransmission timeouts or PAWS dropped SYNs.

Read more on Performance Metric of the Month: CIFS Errors…

Retransmission timeout and round-trip time metrics can reveal virtual packet loss.

This month’s Performance Metric turns to retransmission timeouts (RTOs) and round-trip times, particularly for transactions travelling to virtualized systems. Both high RTOs and high levels of jitter on round-trip times in these instances likely indicate virtual packet loss, a hard-to-detect performance problem that plagues underprovisioned virtual environments.

Read more on Performance Metric of the Month: Packet Loss, Retransmission Timeouts (RTOs) and Round-Trip Times in Virtualized Systems…

This post is written by Eric Thomas, Principal Solutions Architect at ExtraHop Networks.

Many organizations struggle to find the right level of instrumentation to monitor the performance of their business-critical applications. It’s often a compromise: on one hand, you expect a certain level of detail or granularity, and on the other, you know that you will incur some level of overhead—whether in system resource consumption, time spent managing your instrumentation, or both.

Read more on TCP: Where the Network Meets the Application, Part 1…

This post is written by Eric Thomas, Principal Solutions Architect at ExtraHop Networks.

In Part 1 of this two-part series, we imagined the perfect framework for no-impact monitoring of an application stack and found it was right in front of us all along in the Transmission Control Protocol (TCP). Today, we delve into the particulars of TCP measurement, particularly five TCP metrics featured in the analysis of the ExtraHop system: Retransmissions, Retransmission Timeouts (RTOs), Round-Trip Time (RTT), Aborts, Throttling, and Zero Windows.

Read more on TCP: Where the Network Meets the Application, Part 2…

This post is written by Cal Jewell, a Senior Technical Trainer at ExtraHop.

In June, Microsoft announced that it would release an update blocking the use of RSA keys using less than 1024-bit encryption. While the update has been available for testing, the October 9 “Patch Tuesday” will make the update widely available via Windows Update. The update effects all supported versions of Windows, going back to Windows XP SP3. If servers or clients continue to use sub-standard key encryption after October 9, then a range of potential problems could result, including blocked access to SSL-encrypted websites from Internet Explorer, problems installing ActiveX controls and Windows-based applications, and the inability to encrypt or digitally sign e-mail using Microsoft Exchange and Microsoft Outlook. As security experts at the SANS Internet Storm Center point out, this update will not only affect certificates from Microsoft and other well-known authorities, but internal certificate authorities, too.

Read more on Microsoft’s 1024-Bit RSA Key-Size Requirement and Other IT Burdens…

For our November performance metric of the month, we’re looking at HTTP payloads. Hypertext Transfer Protocol (HTTP) is a tremendously important technology layer that supports many critical services. As the basis for web applications and services, HTTP traffic carries a rich trove of information that is important for monitoring application performance, gaining insight into business transactions, and understanding the end-user experience. However, much of this valuable information is unavailable to IT Operations teams with legacy toolsets.

Read more on Performance Metric of the Month: HTTP Payload Details…

The Service Level Agreement is always the first casualty in the war to assign blame.

Most IT organizations monitor their applications and infrastructure in an extremely disjointed manner, with each specialist team relying on tools that provide visibility into a specific technology silo: network tools for the network engineers, database profilers for the DBAs, agent-based APM tools for developers, and so forth. This fractured approach to monitoring contributes to high IT costs, poor user experience, wasted capacity, and an IT organization that is responding to issues reactively instead of proactively.

Read more on Storage Performance Monitoring and the Need for Holistic Visibility…

Unlike IDS/IPS that use signatures, ExtraHop identifies threats by providing context—trends and real historical activity.

Every year, the stakes get higher for IT organizations trying to prevent data theft, ensure compliance, and protect the business from cybervillains. At ExtraHop, we want to help you build a complete IT operational intelligence architecture that provides the visibility you need to protect your environment. The ExtraHop platform can supplement the IDS and IPS tools already implemented with real-time wire data analysis—monitoring of all communications between application components, even encrypted traffic! Whereas traditional security tools are good for securing against known threats, ExtraHop enables IT teams to flag problems and track anomalous events that you would not have known to look for or had the time to look for, including:

• Expired or weak SSL certificates
• Data passing over printer and USB channels in VDI environments
• Access-denied events for networked storage
• High- and low-intensity brute-force attacks on authentication servers
• Data exfiltration through DNS txt records
• Superuser account activity
• … and much more!

All of the scenarios described below also work with our free ExtraHop virtual appliance for IT operational intelligence, so if they sound intriguing, you can get them set up in your environment today. You’ll need to download the the free Security and Compliance bundle, available in the ExtraHop community forum, which will automatically add the requisite dashboard, alerts, and triggers.

1. Encryption Auditing

Managing SSL certificates is a complex and often time-consuming process that can require significant planning, as was illustrated last October when a Microsoft update began blocking RSA keys using less than 1024-bit encryption. Sysadmins simply don’t have the time required to continuously keep tabs on weak SSL keys and expired certificates. As customers begin to phase out weaker 1024-bit keys, ExtraHop can help make a smooth transition by identifying all SSL certificates passing over the network, including those using weak keys and cipher suites. ExtraHop also tracks certificates that are expiring in three months or less, or have already expired, for proactive remediation. This way, not only can you ensure that your encryption is up to standard, but you can prove it with push-button reports.

ExtraHop reveals clients and servers using RSA keys with less than 2,048-bit encryption.

2. Locked-Down VDI Environments

Many IT organizations that run Citrix VDI have locked-down environments to protect and secure sensitive information, like patient health information found in electronic health record systems. In these scenarios, USB and printer channels are locked to prevent both physical and digital data leakage. Because the ExtraHop platform tracks all ICA communications, it can provide continuous monitoring of any data passing over protected channels, with per user and per client drill-downs so you can identify the perpetrator.

3. Storage Access Monitoring

Unlike most transaction monitoring products, ExtraHop analyzes networked storage activity. This enables you to continuously monitor your SAN or NAS environment and break out client IP, username, and file path to identify who is accessing which files from where. By tracking access-denied events, ExtraHop keeps you apprised of any unauthorized users attempting to gain access to secured systems so you can remediate before any damage is done.

4. Authentication Brute Force Alerting

Malicious users may attempt a brute force attack against your authentication services. ExtraHop can detect such attacks in real-time through LDAP analysis. The ExtraHop Security and Compliance bundle is set up to detect both high-intensity and low-intensity attacks by tracking and alerting on the frequency of failed attempts per user and historical counts. High-intensity attacks are akin to smash-and-grab thefts where the attacker tries to gain access and deal damage as quickly as possible before you can react and lock down your environment. Low-intensity attacks try and hide in the noise of regular users’ failed logins, with the hopes that persistent but controlled attempts to access a system will not raise any alarms.

5. Surreptitious Tunneling over DNS

Infected systems can pose serious risks to secured environments, whether they take malicious actions within your systems or exfiltrate sensitive information to external hosts. Several customers have seen surreptitious TCP/IP communications tunneling through DNS, either as a command-and-control protocol for malware or as a method for getting sensitive data back to a home server. By breaking out DNS records by type, and tracking irregular TXT-records and normal A-records specifically, we raise a red flag when potential security events arise, and help you mitigate potential damage.

6. Superuser Account Tracking

To audit any environment, you need visibility into who is accessing your systems. Attackers and rogue applications can use superuser accounts like root and SA to hide their tracks or open security holes. ExtraHop’s Security and Compliance dashboard tracks superuser logins with per client and server IP drill downs so you can quickly take action.

ExtraHop associates superuser account logins with client IPs.

Ready to Get Started? Try ExtraHop for Free!

All the above functionality works with our free virtual appliance for IT operational intelligence, and it’s important to note that these are just examples of the types of security analytics provided by the open and extensible ExtraHop platform. Have ideas for other metrics to track? We welcome your suggestions! In fact, we built the Security and Compliance bundle with the intent that ExtraHop users would extend its functionality and then share their enhancements through the forum.

If you’re interested in reading more about what ExtraHop’s wire data analytics can do for security practitioners, including geomaps and precision packet capture, visit our security and compliance page.

As with the Perl language, ExtraHop makes easy things easy and hard things possible with Application Inspection Triggers, a programmatic interface to the Context and Correlation Engine.

Larry Wall, the creator of the Perl programming language, championed the idea of “making easy things easy and hard things possible.” This month’s Performance Metric of the Month highlights how ExtraHop, with its programmatic interface to its parsing engine, makes hard things possible. In this case, a major web security firm used ExtraHop to pinpoint the cause of extreme latency experienced by a fraction of its users for its web gateway/proxy SaaS offering.

Statistical Averages and Performance Outliers

Web gateways, firewalls, load balancers, content filters, content optimizers, and other proxies break sessions into multiple transactions through network-address translation (NAT) so that A -> C becomes A -> B -> C, for example. This makes it difficult to measure key performance indicators (KPIs) such as latency and processing time per user, which are critical when trying to troubleshoot complaints of slow performance. The web security company mentioned above was receiving complaints from some of its users about extremely slow performance. Even though average performance for all users was great, the IT team knew that the performance of outliers, not statistical averages, held the key to excellent user experience. The trouble was, how to stitch together the transactions broken apart by the NATed proxy?

Challenge: Stitching Together Transactions Broken Up by NATed Proxies

Adding to the challenge, the web security company had no control over the endpoints of the proxy architecture (as would have been the case with a forward or reverse proxy) and could only control the security gateway itself. The IT Operations team considered instrumenting the application and modifying the gateway to insert unique identifier tags. Both options required development work and added complexity and cost—possibly hundreds of thousands of dollars depending on the transaction tracing tool used. Compared to these options, the solution with ExtraHop was trivial. Using Application Inspection Triggers, the programmatic interface to ExtraHop’s Context and Correlation Engine, the web security company created unique session fingerprints and stitched together HTTP flows so that they could measure each leg of every transaction, even as those transactions traversed a complicated proxy architecture.

The web security company needed to measure latency for each stage of the full transaction, but did not have control over the ingress and egress points, only the web gateway proxy.

To build a unique identifier for the request flow (1-7 in the diagram above), the team built an Application Inspection Trigger that would recognize the URI, UserAgent, and the client IP contained in the X-Forwarded-For HTTP header field. For the response flow (4-11 in the diagram above), the team built the unique identifier from the URI, proxy IP address, ETag header, Set-Cookie header, and Expires header. Together, these identifiers comprised a unique fingerprint for each transaction that did not depend on instrumentation of the application code or inserted tags. It’s worth noting that this type of agentless recognize-and-trace transaction tracing is even simpler in scenarios without a proxy. In those cases, IT teams can use a existing unique identifier such as a customer ID, Object ID, or the embedded tags and session IDs inserted by JSP, PHP, and Microsoft ASP. For example, ExtraHop offers a solution bundle that recognizes and traces multi-hop web-to-database transactions for SharePoint. The point of this web gateway illustration is to show the extensibility of ExtraHop wire data analysis to handle the worst-case scenario, or to put it in terms that Larry Wall would appreciate, to make easy things easy and hard things possible.

Solving Complex Performance Problems

By building a unique fingerprint for every transaction in ExtraHop, the IT Operations team at the web security company was able to answer the following questions:

• What was the latency for the complete transaction (1-11 in the diagram above)?
• What was the latency for the request across the proxy (2-6 in the diagram)?
• What was the latency for retrieving content from the destination (7-8 in the diagram)?
• What was the latency for the response across the proxy (9-10 in the diagram)?

Median and percentile bands provide a good picture of overall performance, but can hide important statistical outliers.

In ExtraHop, the IT Operations team could see the median latency and 25^th to 75^th percentile spread for each leg of the complete transaction, shown in the graph above. These averages showed that performance was good. Viewing the ExtraHop metrics in a heatmap, however, revealed 95 percentile outliers all the way to the 9-second mark on responses traversing the web gateway proxy. This indicated that the proxy itself was introducing the delay. With the help of the development team, the IT Operations team found a DNS reverse-lookup process that was timing out. Fixing this process eliminated the unusual latency (as much as 2 minutes!) experienced by some users.

Extensibility Is Important

While other IT monitoring solutions can do a few prescribed tasks well, ExtraHop offers a programmatic interface to its Context and Correlation Engine that enables IT teams to tackle unexpected challenges, such as transaction tracing across a NATed proxy such as a gateway, firewall, load balancer, content filter, or optimizer. As far as we know, there is no other solution that can do this without instrumenting the application or inserting tags. Would you like to try out the extensible ExtraHop platform for yourself? You can by downloading the free ExtraHop Discovery Edition virtual appliance and exploring the solution bundles on our forum. The transaction tracing solution bundle for NATed proxies, which provides the capabilities described above, is also available for download in the ExtraHop forum.

Watch the video below to learn about the importance of investigating anomalies and outliers in your datasets. Learn more about how ExtraHop’s visualizations preserve meaning when aggregating large sets of wire data.

Most enterprise IT organizations use application delivery controllers (ADCs) to improve the availability, speed, and efficiency of the IT infrastructure. However, ADCs offer limited metrics to help in tuning performance or troubleshooting problems. ExtraHop’s co-founders led the design of the BIG-IP v9 product and TMOS platform at F5 Networks, so we are perhaps more cognizant than most about the things to look for when tuning and troubleshooting application delivery controller (ADC) performance.

Because the ExtraHop platform analyzes all L2-L7 communications on the wire, it serves as a great way to evaluate the effectiveness of ADC policies and settings such as HTTP compression. We often hear back from customers who have dramatically improved their ADC performance due to the insights gained from their ExtraHop deployment, as in the example below.

Citrix Delivery + Web Services + ADC = Complexity

One large IT services firm used ExtraHop to gain insight into how well its ADC was supporting a major sales productivity application. This particular application was delivered to a distributed sales team over Citrix published desktops. At first, the Citrix tier was suspect, but ExtraHop showed that it was not to blame in this case. Read more about ExtraHop’s ICA monitoring and analysis, including login and application launch times per user.

Analyzing HTTP Caching and HTTP Compression

Next, the IT team used ExtraHop to investigate the web services part of the application that pulled data from an AS400 as well as other enterprise applications. A virtual server on an F5 BIG-IP Local Traffic Manager load balanced the web services requests (in the form of HTTP POSTs) across a number of back-end nodes. Importantly, each system had limited 100Mbit connections to one another. When viewing the HTTP analysis, the IT team realized that average response sizes were between 3MB and 7MB, with some responses exceeding 32MB. For 100Mbit links, these large chunks would add significant latency to the application.

Large HTTP response sizes add signifiant latency over 100Mbit links.

ExtraHop also revealed that virtually all the MIME-types were text-xml and that there were no compressed transfers. The upstream clients did support compression, so there was a very strong case that this was what was needed to improve end-user experience.

ExtraHop revealed that compression was turned off.

Testing the Effectiveness of ADC Changes

Previous to deploying ExtraHop, this IT organization had few options for evaluating and monitoring ADC performance except for a best-guess, wait-and-see approach to testing, which was made especially unappealing because of a restrictive change-management process. ExtraHop made investigation much easier by revealing what type of content was being delivered and how well. With this type of visibility, optimizing the ADC was a much more scientific and effective endeavor. The IT team turned on compression for a staged virtual server, and could see in ExtraHop the change that they were looking for with compressed transfers increasing and smaller response sizes.

Complex Apps Giving You Trouble? Bring in ExtraHop

Traditional monitoring tools cannot solve the type of problem described above. First, there are no other completely passive solutions that can tie the latency of specific Citrix sessions to activity on the backend as ExtraHop can. Second, traditional agent-based monitoring solutions are blind to ADC performance. Start analyzing wire data in your environment today with the ExtraHop Discovery Edition, a free virtual appliance.

If you’re interested in learning more tips and tricks to optimizing your ADC performance, check out the Data Center Journal article below from ExtraHop CEO Jesse Rothstein, who previously led the design of the BIG-IP v9 product.

If you are responsible for secure web-based services, it is likely that you are scrambling to identify servers using OpenSSL versions 1.0.1 through 1.0.1f, trying to patch those servers, and reissuing certificates.

ExtraHop can detect the heartbeats that are used in the Heartbleed exploit, revealing potential attacks against your SSL servers. This capability is available in the ExtraHop Discovery Edition, a free-forever virtual appliance.

Follow these steps to install the Discovery Edition and detect Heartbleed exploits:

1. Fill out the form at www.extrahop.com/discovery
2. Receive your product key and download the .ova file
3. Install the virtual appliance on a machine with a bare-metal hypervisor or in AWS
4. Direct traffic to the appliance using our software tap or a port mirror
5. Install the Heartbleed solution module (a one-minute process)

Note: If you already have a full version of ExtraHop deployed, you do not need to download anything extra to do the analysis described below, although we have bundles in the forum that provide custom pages and geomaps specifically for Heartbleed activity.

The free ExtraHop Discovery Edition shows potential Heartbleed exploits.

How ExtraHop Detects Potential Heartbleed Attacks

ExtraHop performs detailed SSL transaction analysis: certificates used, session details, cipher suites, connections over time, record sizes, and other metrics for every SSL transaction. We also break down SSL records by content type, including application data, change cipher, handshakes, and alerts. There was one other content type that we did not list out by name in the user interface because it was an extension and not part of TLS core … You guessed it, the last content type we record is heartbeat, which is the message used in the Heartbleed exploit. In the ExtraHop user interface, heartbeat messages currently appear as “Other.”

ExtraHop breaks out SSL transactions by content type. View the TLS content type registry, including the now-infamous heartbeat.

The heartbeat content type was previously obscure and rarely used, which means that any SSL traffic using heartbeats is worth investigating. Investigating heartbeats is a simple drill-down in ExtraHop. First, you navigate to the SSL Server activity group.

From the SSL Server activity group, we can see records by content type. Clicking on “Other” will show server devices that have received heartbeat records.

From there, you can drill-down investigate each SSL server to see which clients are sending heartbeat messages. If you don’t recognize a device and it is sending many heartbeats, then that is a potential active exploit. (Note: Device-level L4-L7 views are not available in the Discovery Edition, which is a great reason to upgrade to the full version at this point!)

But there is more that ExtraHop reveals about potential Heartbleed attacks. With the ExtraHop geomap capability (available in the Discovery Edition) you can see the geographic origin of requests for a particular protocol with geomaps. The screenshot below comes from a retailer using ExtraHop to do just that. They quickly saw that heartbeat messages were originating from St. Petersburg, Kiev, Chengdu, Wuhan, and other places that are highly suspicious.

Don’t delay. Request your own ExtraHop Discovery Edition now and discover the power of wire data analytics!

ExtraHop geomaps reveal the geographic origin of requests. This geomap is from an IT organization that uses ExtraHop and shows six heartbeat messages originated in Kiev, Ukraine.

Read more: DevOps Hearts Race While CISO Looks for Heartbleed

Vincent Yesue is a member of the ExtraHop Solutions Architecture team.

In the retail industry, slow web application performance means abandoned shopping carts online, and long lines and frustrated customers in stores as retail employees use those web applications to check inventory and process transactions.

In the video below, ExtraHop Solutions Architect Vincent Yesue explains how ExtraHop helped dramatically speed up one retailer’s in-store web applications, bringing tears of joy to a store manager. The ExtraHop wire data analytics platform helped the retailer identify and resolve two issues slowing their web applications:

1. A web proxy auto-discovery (WPAD) setting that was enabled on browsers but not on the Active Directory servers. Disabling WPAD on all browsers eliminated a 40-second delay when launching a new session.
2. Router misconfigurations in the stores meant that public Internet traffic (comprised of YouTube, Instagram, etc.) was consuming bandwidth that was supposed to be prioritized for application data. Updating the routing profile ensured that only web application data was encapsulated and sent back to the datacenter for processing, and that public Internet traffic was sent directly to the relevant sites.

Read more about ExtraHop’s professional services offerings and how national gas station and convenience store chain Murphy USA uses ExtraHop to optimize retail web applications.

Video transcript:

One of my customers called me a short time ago. This customer is a nationwide retailer that has outlets all over the country. … As we examined the wire data, we found that some requests were being from that web browser, on the first request after it started up, for a Windows [Web] Proxy Auto Discovery feature that wasn’t enabled on their network.

… We readied a configuration change and actually made a call back to the store that we had visited earlier in the day. We got one of the retail personnel on the line and asked her to bring up the application and go through the workflow that she would normally suffer with, and lo and behold she was still having problems. Things we were still slow even though we had made that WPAD proxy discovery change—she was still struggling.

So we told her to hold on just a second. We made the change to the routing in the store and we asked her to load the application again, and she started crying. The application was just blazing fast now and the problem that they were fearing going into Christmas with these long lines and being unable to service transactions very quickly was now not going to be a problem at all because we had solved the problem and these applications began to be really, really fast—just like they should be.

There’s some World Cup mania taking hold at ExtraHop HQ, as is likely in offices around the world. Our technical marketing engineering (TME) team expressed their enthusiasm in a unique way—by building a solution module for monitoring World Cup streaming with ExtraHop. The module works with the free ExtraHop Discovery Edition so you can try it out for yourself!

With the World Cup Tracker module, you can see:

The World Cup Tracker module parses web requests to ESPN Gamecast to pair match IDs in the header with match names in the payload.

• Visits from clients on your network to various websites offering World Cup content, including ESPN.com and FIFA.com, along with the client IP addresses and device names.
• The amount of web traffic requested from each site, along with client details, providing insight into the amount of World Cup streaming on your network.
• Which ESPN Gamecast matches are most popular in your office. For example, you can see Brazil vs. Croatia was viewed more often than Switzerland vs. Ecuador.

Platform Benefits: Build, Extend, Share

The World Cup Tracker module is another great example of the flexibility of the ExtraHop platform. You can easily program the real-time stream processor at the core of the ExtraHop platform to analyze the L2-L7 communications between servers however you want. An example: To show which matches people are viewing on ESPN Gamecast, the TME team paired the unique match ID in the header (383303) with match title contained in the HTTP payload (Brazil vs. Croatia). With the ability to mine anything in the transaction payload, the possibilities are really only limited by your creativity!

Ready to get started? Download our free ExtraHop Discovery Edition virtual appliance, then apply the World Cup Tracker module. While you’re at it, you might also want to check out our cloud monitoring bundle that lets you track usage of cloud applications such as Google Drive, Dropbox, Salesforce, Spotify, and others.

ExtraHop parses transactions to World Cup streaming sites so you can track streaming activity.

Tips for Handling World Cup Streaming

From a technology standpoint, the 2014 World Cup is much different than 2010. Online video streaming is much more common and matches are frequently live during the workday in the United States. Here are some tips for dealing with World Cup streaming at your office:

• Set up streaming in a conference room so that employees are not eating up bandwidth by watching games at their desks. It’s more social, too.
• If your organization has a policy regarding streaming, communicate that with employees now.
• Security experts warn that the World Cup is prime-time for hackers. Warn your users against visiting dodgy sites and instead stick with legitimate sites like ESPN.com and FIFA.com.

Troubleshooting poor web application performance can be frustrating because you don’t know where to look first. That’s why we created the web application performance troubleshooting guide (no registration required!). It shows how to troubleshoot common web application performance issues using the free ExtraHop Discovery Edition. By analyzing your wire data—all L2-L7 communications on the wire—you can quickly troubleshoot issues caused by overloaded servers, changed APIs, crashed equipment, network bandwidth constraints, and more.

Sometimes the Network Really Is the Problem

So let’s get to it. In this post we are focusing on network connectivity and performance problems. “The network is slow” is an all-too-common lament. Despite its frequent vilification, it is worth the time to investigate how well the network is delivering applications. Where should you start? The ExtraHop Discovery Edition makes this investigation easy by showing slow performance or anomalous network activity caused by unplugged cables, broadcast storms, rebooted network switches, virtual packet loss, or modified VLAN tags, for instance.

The steps below demonstrate how you can identify a low-level network connectivity issue. And while this scenario primarily relies on simple L2 analysis, keep in mind that ExtraHop also provides terrific L4 TCP analysis and L7 application-level analysis so that you can dig deeper into web application performance issues.

Looking at the Summary page, we can see what appears to be two minutes (14:47 to 14:49) of almost no activity on the network.

Selecting Network and then L2 in the tree control, we can verify that both the number of packets per second and throughput dropped to near zero (12.73 packets/second and 12Kbps, respectively) during that time period. Most likely, this trickle of traffic is localized broadcast traffic.

Layer 2 traffic was virtually nil for two minutes, indicating a network equipment problem. It seems likely that a network switch rebooted, preventing users from connecting to the web server. Looking at Layer 2 activity helped to quickly identify this issue. Now, this leads to the next question, which is why the network switch rebooted. We’ll tackle that question in the next scenario. Stay tuned for next week’s post!

There are many other situations where ExtraHop’s wire data analytics can help to make life easier. Ready to give it a try? Download the free ExtraHop Discovery Edition today and start analyzing your wire data.

The ExtraHop Discovery Edition is a virtual appliance you can run on a bare-metal hypervisor, on a desktop hypervisor, or in the AWS cloud. It provides you with the ability to listen in on what your servers are communicating to each other on the wire.

Welcome back to our Web Application Troubleshooting series, where we walk through how to diagnose common web application performance issues including network switch reboots and devices soaking up bandwidth. In this post, we’re going to solve another web application performance problem, an overloaded server, using wire data analytics.

You might use some sort of basic server monitoring already, but there can still be delays between the time a server crashes and when the IT team is notified. More commonly, a web or database server doesn’t crash but just responds to requests slowly because of heavy load. By observing the communications between servers, you can quickly detect crashed or overloaded servers and fix the problem.

Suppose a database server is shared among several other applications in addition to your web application, and a development team member pushes out an ill-advised bit of code that soaks up the database server CPU resources. Now your web application is responding slowly, but how can you find out the root cause? Let’s look at how you’d go about doing this type of investigation using the Discovery Edition.

Eavesdropping on Your Web Server’s Conversations

When trying to identify a crashed or overloaded server using the ExtraHop Discovery Edition, the best place to start is the Summary page, which displays a high-level overview of activity at various tiers of your web application stack, including network, web, database, storage, authentication, and domain name services.

From the Summary page, you want to look for abnormalities in activity. Make sure to use the Time Interval options to make variances more apparent. In this scenario, we have noticed a suspicious dip in traffic. Narrowing the time interval, we can see that the volume of HTTP traffic suddenly disappears for a period of several minutes. To investigate, we click HTTP on the Bytes by L7 Protocol chart.

Here, we see a list of all devices communicating using the HTTP protocol. The first two devices in the list stand out because their volume of HTTP traffic is significantly higher than the others, and also because their inbound and outbound traffic counts are inversely correlated. One client is requesting 70GB of HTTP data from a web server. There is definitely something suspicious going on here.

Clicking the first device—the one requesting 70 GB of HTTP data—pulls up metrics specific to that device.

Click the Protocol Breakdown tab and hover over the chart to see the exact traffic measurements for each L7 protocol at specified times. We can see that even when HTTP traffic dropped off, this device was still online and communicating, receiving over 7MB of HTTP traffic. Because this device has been functioning during the time period under investigation, we close this window and look at the web server that is sending 70GB of HTTP data during this time period. Clicking the Protocol Breakdown tab for the web server tells a different story.

During the same time period, the web server stopped communicating not only over HTTP but all other protocols as well. This indicates that the device lost network connectivity, rebooted, or crashed. In any case, by viewing L2-L7 communications for all systems, we were able to quickly isolate the problem server with just a few clicks.

Note: The ExtraHop Discovery Edition shows summary metrics for specific devices, but with the full ExtraHop Enterprise Edition, you can drill down to see transaction details for specific devices such as error counts, methods used, users, and files accessed.

In summary, a web server lost connectivity, rebooted, or crashed for several minutes. We could very quickly diagnose the problem with visibility into application-layer (Layer 7) communications on the wire. This view not only identifies the server experiencing problems, but also applications or users that might be causing a problem.

Be Prepared—Listen to Your Wire Data

So there you have it. The free ExtraHop Discovery Edition provides you with the ability to listen in on what your servers are communicating to each other on the wire. Our Web Application Troubleshooting Guide shows how to use the Discovery Edition to troubleshoot common performance issues. What are you waiting for? Download the free virtual appliance and guide today so that you’re prepared for the next inevitable web application issue. If you’re not ready to download the virtual appliance yet, you can always kick the tires with our free online demo (no sign-up required!).

Stay tuned for our next post on misconfigurations and application errors.

Following on the heels of the Heartbleed and Shellshock exploits, the new POODLE vulnerability in SSL version 3.0 (SSLv3) is the latest to require IT teams to identify and patch vulnerable systems.

Published by Google’s security team today (Tuesday, October 14), the POODLE vulnerability targets a version of SSL that is 15 years old but still used widely. IT teams will want to identify systems using this version and disable SSLv3 on those machines if possible.

At ExtraHop, identifying vulnerable machines was a 15-second process (see the screenshots below for the results). That’s because we have an ExtraHop appliance analyzing all our wire data—all L2-L7 communications between systems—and extracting a wealth of information for easy exploration. Whether it is identifying devices using SSLv3 or performing a Heartbleed audit going back years, ExtraHop puts your wire data at your fingertips.

Identifying SSLv3 Servers and Clients in Four Clicks

If you are an ExtraHop user, here is what you need to do in order to identify SSLv3 sessions in your environment:

1. Click on the Applications tab in the left-hand navigation
2. Click on the “All Activity” application
3. Click SSL in the left-hand navigation to view all SSL metrics
4. Click on the SSLv3 count under Sessions by Version

The resulting window will show you the top talkers for SSLv3 in your environment—these are the systems you will want to update first. Note that exploiting the POODLE vulnerability requires a lot of chattiness. Adjust the time interval to see more devices. You can also see the clients and certificates involved in these sessions. This capability is available for both our Enterprise Edition and Discovery Edition. If you want to add a nifty dashboard that visualizes these SSLv3 metrics, download the bundle from the Send The POODLE To The Pound forum post.

This is just one example of what you can do with wire data. The possibilities are virtually limitless! Find out for yourself by exploring our online Enterprise Edition demo.

If you do not have an ExtraHop yet but need to identify devices using SSLv3, this is a great time to download our free-forever virtual appliance, the ExtraHop Discovery Edition!

ExtraHop’s SSL envelope analysis reveals all kinds of interesting details about encryption in your environment, including SSLv3 usage.

Drilling into SSLv3 conversations, you can easily identify top-talkers using that version.

Adding a widget showing SSLv3 top-talkers to your dashboard is a simple three-step process.

Roughly 1,000 retail companies have been impacted by the Backoff point-of-sale malware with costs related to data breaches totaling more than $150 million, according to a recent eWEEK article. If your organization relies on point-of-sale terminals, the best approach is to assume that your systems are already compromised and to look for malicious activity within your network.

Read the U.S. CERT advisory on Backoff, including capabilities and mechanisms.

To help IT organizations in this effort, I have created a bundle (a simple extension to the ExtraHop platform) that detects and tracks the HTTP-based command-and-control (C&C) traffic for the Backoff malware family. I used the code snippet provided by SpiderLabs to write the Application Inspection Trigger included in the bundle. The trigger parses the HTTP payload and detects the RC4 and MD5 data fields used to encrypt stolen data and hash the password. This bundle cannot be guaranteed to detect every Backoff variant, but should detect a majority of the variants. The bundle also includes an alert and a custom dashboard for tracking infected systems, IP addresses of C&C servers, and C&C messages.

Why ExtraHop’s Solution Is Unique

Backoff malware uses HTTP for command-and-control communications, such as this HTTP POST message with stolen data encrypted.

The ExtraHop solution for detecting Backoff malware can be implemented in minutes, requires no agents, and will not affect production systems apart from the ExtraHop appliance. Moreover, once Backoff activity is identified, the ExtraHop platform provides an excellent source of data for forensic investigation, enabling you to understand the context of the infiltration, including which systems are involved and what data was targeted for exfiltration.

For these reasons, the ExtraHop platform is an excellent complement to traditional methods of detecting malware such as Backoff, namely, antimalware software running on end points and inline intrusion detection systems (IDS) that rely on vendor-provided malware signatures.

You can download the bundle now from the ExtraHop community forum. If you see anything that can be improved, please feel free to add that in the comments. This is a great example of the power and flexibility of the ExtraHop platform to empower your teams with operational intelligence.

If you don’t have an ExtraHop appliance yet, but would like to try the Backoff detection bundle, you can get started by requesting your own free ExtraHop Discovery Edition, a perpetually licensed virtual appliance.