When it Comes to Alarm Management, Don’t Reinvent the Wheel

By Hector Perez, Director of Business Development & Technical Consulting at PAS

Stone Wheel

I recently presented at the Ocean Energy Safety Institute Seminar on Focusing on Alarm Management for Safer Offshore Operations. It was exciting to hear what peers in other industries had to say regarding their approach to alarm management. The panel included professors from Texas A&M University and representatives from Schlumberger, NASA, and the chemical, petrochemical, and power industries.

At the end of the interactive day, I concluded that the alarm problem is exactly the same for all industries: we are generating too many alarms for console operators. By too many, I mean the quantity of alarms is not manageable and the operator must ignore some of them. Of course, this is not by choice, but rather due to human limitations in cognitive loading. Ignoring alarms, as we all know, is a dangerous task where sooner rather than later the “wrong” alarm is ignored and severe consequences happen.

In other words, when approaching the alarm management problem, we don’t need to spend time reinventing the wheel. Through my 15+ year career in the energy sector, time and time again I have heard representatives from each of the different industries say things such as, “we are different than X industry; we are Y industry” or “our alarm problem must be different than theirs.” What I’ve seen across industry lines is too many alarms, and on top of that they are incorrectly prioritized.

Much investment has been made in both time and money to practically achieve sustainable improvements for alarm systems. Books have been written on it. PAS’ The Alarm Management Handbook provides a field-proven methodology for approaching the problem. Much of the information available today is applicable cross-industry, so learn from it and then decide how exactly to apply it to your specific needs.

Is your company trying to reinvent the alarm management wheel?

Innovation Convergence: The Future is Now

By Tamara Anderson, VP of Corporate Strategy and General Counsel at PAS

IdeaI love hearing from tech leaders who live and work on the bleeding edge. Topics that the rest of us consider futuristic are addressed by tech pioneers in the context of here and now – just as necessary and commonplace as prioritizing your to-do list or choosing a restaurant for tonight’s dinner. At a technology law conference in Austin last week, leaders from companies like Tesla, Rackspace, and Dell highlighted how developments like 3-D printing, genome editing tools, and autonomous vehicles impact business and culture. My key takeaway: the future is now, and it’s leaving a lot of slow-moving companies in the past at ever-increasing speeds.

The idea that my 16-year old will be among the last of humans who will learn to drive a personal automobile is startling. Shared self-driving fleets will safely, efficiently and inexpensively transport the next generation to their destinations, eliminating the 23 hour per day inefficiencies of most privately owned cars. Examining the social and microeconomic implications of such collaborative consumption in sectors like transportation, co-working, and tools is even more astonishing.

We’ve all benefitted from the exponential growth in computer processing power in ways even futurists couldn’t have imagined a generation ago. Similarly, ancillary technologies have experienced Moore’s law in tandem, enabling a convergence of collaboration and innovation across virtually every industry. For those of us in the Industrial Control System (ICS) industry who straddle the contradictory worlds between ubiquitous connectivity and 30-year-old automation systems, we must ask, “is the ICS industry keeping pace?”

Arguably, the process industries have not benefitted relative to their importance in our economy and lives. We’ve seen vast improvements in sensors and devices for gathering data. We’ve significantly enhanced production and asset performance. But we are just starting to address the convergence of Information Technology (IT) and Operational Technology (OT), standardization and opening of ICS architecture, and securing of proprietary automation assets from cyber threats. As an example of this, in a February blog entry I wrote about ExxonMobil’s collaborative approach to directing innovative change in ICS architecture, a move that would finally allow ICSs to leverage beneficial technologies and best practices.

The Fourth Industrial Revolution is upon us. It is changing the way we live, work, and relate to others, as though the last few decades haven’t been transformative enough. Ushered in by The Internet of Things, the velocity of disruption is creating whiplash across our cautious, slow-adopter ICS industry. Even so, data visualization, predictive analytics, and machine learning are as relevant to plant operators and security officers as mapping the human genome is to a geneticist. Forward-thinking companies such as PAS leverage these technologies to access and optimize big data and operational information in the plant environment, redefining operational, safety, and security outcomes. We can’t afford to allow our critical infrastructure cyber-physical systems to lag behind technological innovation, because threats to plant reliability, safety, and security won’t wait for us.

Are We Four Lines of Python Away from Cybersecurity Trouble?

FourBy David Zahn, CMO and GM of the Cybersecurity Business Unit at PAS

A USA Today reporter recently interviewed the guys who hacked the Jeep Cherokee last year. One of the white hats said something that was particularly distressing. He said that he wrote “four lines of python and owned [had access to] 1.4 million cars.” What gets me is not that he did it (he revealed their hack to Fiat/Chrysler before publishing), but that a hunk of metal traveling down the road at 60 or 70 miles an hour – something in which I transport my children – was manufactured without any real thought to cybersecurity. If cybersecurity was a design consideration, then surely it would take more than four lines of code to get into the car’s systems.

Unfortunately, car manufacturers are not alone in their myopic view of cybersecurity. I was recently on a plane heading to San Francisco to speak about the very topic of cybersecurity at the AFPM (American Fuel & Petrochemical Manufacturers) Annual meeting. As I looked up from my seat, I wondered if there were any hackers on board who were willing to break into the cockpit control systems as was allegedly done last year on a flight. Why have we not adopted a better cybersecurity approach within our manufacturing sector? Please tell me we don’t need a hack similar in magnitude to the Target one (but instead of financial losses we suffer lives lost) before companies take this seriously.

Of course, I’m not the only one with this concern. We hear it voiced in our conversations with every customer and at every conference. At the 2016 ARC Industry Forum, there was a lengthy panel discussion about where to invest in ICS cybersecurity when most risk scenarios are characterized as low risk but high impact. PAS founder and CEO Eddie Habibi rightly said you need to approach these just as we have always done so – by assessing ICS cybersecurity from a safety and risk perspective. When you look at the individual risk scenarios in aggregate, it is difficult to argue an approach that ignores applying basic principles of cybersecurity such as inventory, patch, and configuration management for the proprietary control systems (where a built-in cybersecurity approach to 10 to 25 year old systems just is not an option). It is clear that such measures directly address the risks presented by malicious attacks – not to mention ones from engineering mistakes. With safety absolutely in the crosshairs, how much more time do we want to give the bad guys targeting industrial control systems to write their version of “four lines of python?”

When ICS Cybersecurity Gets Personal: The Risk of the Disgruntled Employee

CybersecurityBy Nick Cappi, Director, Global Business Development for Integrity Solutions at PAS

In recent news, we’ve learned that a former Georgia-Pacific IT specialist is now facing prison time and significant fines following the cyber attack he waged upon his ex-employer – an attack which occurred just days after he was fired by the company. He caused disruption at the Port Hudson paper mill targeting the distributed control system (DCS) and quality control system for the machinery used to produce paper. It took Georgia-Pacific personnel significant time to evaluate and understand all changes made to the plant’s control systems’ programs.

This story really touches a nerve for me as a deliberate attack from a past or present colleague is personal. This wasn’t the first time a disgruntled employee tried to harm his former employer, and it won’t be the last. The stakes are certainly higher now, but this is unfortunately the world in which we live. Challenging economic times, such as the current oil and gas downturn, are causing employees to lose their jobs, which can generate ill will towards former employers.

Our own conversations with Chief Information Security Officers (CISOs) demonstrate that the disgruntled employee is a growing concern and is in fact part of an even greater corporate cybersecurity concern that rises to the Board of Directors level. The U.S. Federal Bureau of Investigation has reported annual losses of more than $800 million dollars in the U.S. due to cyber crime. Beyond the financial impact, companies risk lost production time, regulatory penalties, brand equity impacts, director-level liability, and safety and environmental losses.

Keep in mind also that existing employees, those meaning no harm, also present a risk. I have worked in the industrial control space for the past 20 years. During that time, I have seen numerous unintentional changes to control systems that have caused similar damage as the Georgia Pacific cyber incident. Good configuration management from field instrumentation through the HMI visualization layer including execution logic is critical to having safe and secure operations regardless of the intentions of the individual making changes to those systems.

We speak regularly on these topics with customers and at conferences, and it’s important to consider mitigation steps in the face of the ongoing affects that the global oil and gas market downturn and potential future layoffs will continue to have. Based on industry lessons learned, two things must be considered:

  • Detection: Part of a holistic change management process is early identification of these unauthorized changes. If these changes were not part of a managed process, site personnel should be alerted immediately. Early detection would have been key for Georgia-Pacific and many others.
  • Recovery: Beyond detection, monitoring policy violations and automating workflow-driven responses are needed. It took a significant amount of time for Georgia-Pacific resources to identify each aspect of damage sustained from the attack and additional time to correct the problems. Identifying exactly what changed and how it was configured before the attack takes just minutes with change management automation. This is the key to a quick recovery.

It’s much more exciting to talk and write about external threats from rogue states and terrorist organizations who threaten to take control of our critical infrastructure compared to some control engineer (with or without malice) making a change that takes down the same critical infrastructure. When all the hype is over, what do you think has the greatest potential risk and probability of happening at your site?

Cybersecurity Road Trip

By David Zahn, CMO and GM of the Cybersecurity Business Unit at PAS

suitcaseDuring the last two weeks, I had the privilege to speak on cybersecurity at two industry conferences, the ARC World Industry Forum and the SANS 11th Annual ICS Security Summit. While these represent two different audiences, the sessions and conversations were strikingly similar and I think reflective of how industry is viewing ICS cybersecurity. Here are some of the consistent themes:

You must know what you have. Users and vendors alike emphasize the need for an accurate inventory as the first step in an ICS cybersecurity strategy. Managing IT-based cyber assets like workstations and routers is fairly straightforward (and there are numerous solutions on the market that address this need). However, proprietary industrial control systems are too often blind spots for plant personnel. The sentiment is that the most widely used solution for inventory today – spreadsheets – is not scalable or robust enough to track the inventory and configuration data essential for an effective ICS cybersecurity strategy.

The waters of ICS cybersecurity are getting muddier: More and more vendors are entering the ICS cybersecurity market or are touting ICS cybersecurity capabilities. The growing din of solutions is making strategic decision making more complicated for companies. I was encouraged to see ARC propose a maturity model that prescribed capabilities companies need to become more secure. SANS takes a different approach with their model by describing how to detect and block attacks at various stages/steps within the kill chain. Both are effective in helping companies gain clarity on where to focus their cybersecurity hardening efforts. The more analysts and best practice clearinghouses such as ARC and SANS can do to cut through the noise and help companies evaluate adoption opportunities, the more secure we become as an industry.

Cybersecurity is a priority, but there is greater budget scrutiny: With the downturn in oil and gas, corporate budgets continue to shrink. Although cybersecurity tends to remain well-funded, companies are applying greater scrutiny to every dollar spent and are demanding better articulation of value from technology vendors. As a solution provider, we address this with our cybersecurity offering and can show demonstrable gains in engineering productivity, reduced compliance and operational costs, and improved asset reliability and safety.

ICS cybersecurity continues to gain traction with IT and OT organizations. Companies are not waiting for an incident or regulation to take action, and are looking for standards-based technology solutions that address the gambit of cyber assets found in plants today. Conferences like ARC and SANS provide a valuable forum for industry peers to exchange ideas and keep a pulse on evolving cybersecurity standards and technology.





Innovation Required: ExxonMobil Drives Revolutionary Change in ICS Industry

By Tamara Anderson, VP of Corporate Strategy and General Counsel at PAS


ExxonMobil’s Research and Engineering Company (EMRE) is demonstrating timely leadership through its initiative to re-architect the process automation platform. On January 14, 2016, EMRE announced the award of a research-phase contract to Lockheed Martin (LM) to champion the design of an open and secure next-gen architecture emulating the open avionics environment embraced by defense and aerospace industries.

End users in the industry will welcome this initiative, as we’ve heard them cite countless reasons to reform automation systems, including: interoperability challenges, cost of integration and maintenance, obsolescence, and high cost of migration. The emergence of the Industrial Internet of Things, the availability of astonishing computing power, and the requirement for an intrinsically secure platform, have all converged to compel an urgent opportunity for disruptive change in the Industrial Control System (ICS) market. Despite the commoditization and standardization of IT hardware over the past 30 years, OT systems remain largely proprietary and expensive. Factors that historically justified the status quo – safety, cost, ICS evolution process – have been eclipsed by technologically advanced solutions to those very challenges. Similarly, advances in mission-critical software for cybersecurity, data management, and predictive analytics, require openness and interoperability to be fully and cost-effectively utilized.

LM, known for its strong systems engineering capabilities, particularly in requirements development, seems a good partnering choice to drive EMRE’s vision for next-gen automation systems. Ironically, the LM-led development of the F-35 Lightning II (winning design of the Joint Strike Fighter program) has been plagued by cost overruns and delays attributed in large part to proprietary hardware and software problems.

In an article posted on the U.S. Army website, author Bill Crawford quotes AMRDEC’s Alex Boydston while discussing common architecture efforts. Boydston states that, “Seventy percent of new aircraft development cost is now in software.” The Pentagon has called on LM and the F-35 program office to embrace interoperability and to allow future upgrades to the F-35’s avionics software to be open to other vendors. LM is a sponsor of the future airborne capability environment (FACE) consortium, the Open Group working with the F-35 program, and others to develop software standards, business strategies and certification processes to define and promote an open avionics environment. In fact, the FACE model is presented by EMRE and LM as the success example on which they intend to build their ICS model.

Don Bartusiak, Chief Engineer at EMRE, explained at the Industry Day hosted by LM last week, that ExxonMobil will replace a significant percentage of its control systems over the next decade. Dr. Bartusiak will have more to say next week in Orlando at the ARC Industry Forum. Although EMRE’s vision is for an industry-wide solution, it is clear that ExxonMobil intends to shepherd the transition from a proprietary stovepipe model to a full open ICS architecture. If they are successful, opportunities will abound for agile and innovative companies like PAS. And if LM also achieves an open interface to the F-35’s avionics, then refineries and strike fighters will indeed share some significant innovation success.

Do you think this is an opportunity for fast-moving innovators?

Aliens and Cybersecurity?

By David Zahn, CMO and GM of the Cybersecurity Business Unit at PAS


In 1964, Nickolai Kardashev proposed a classification system – the Kardashev Scale – for identifying how advanced a civilization is based on its ability to harness and use power. In this system, a Type I civilization has the ability to harness the sun’s energy that reaches Earth, a Type II has the ability to harness the entire power of the sun, and a Type III can harness the power of the galaxy. Other scientists have since extended the system to describe civilizations in finer detail. Under these extensions, we are living today in a Type 0 civilization in which the power we harness comes primarily from oil, gas, coal, and other non-renewables. Futurists predict we will achieve Type I status in roughly 100 years.

Why is this important, and what does this have to do with aliens? As civilizations transition from Type 0 to Type I, their ability to destroy themselves increases. Geopolitical conflicts, extremism, and other causes of violence have greater consequence when energy reserves at such scale are available to do harm. This potential is often used as an explanation as to why we have not seen aliens to date. Most civilizations simply do not make it to Type I.

While this sounds like science fiction, there is some sense to it. As we approach Type I status and develop technologies that increase our mastery over the sun’s energy, it is essential that we keep that power under lock and key. This means keeping critical infrastructure safe and secure from malicious attack or accident. Unfortunately, we are not even close to this level of protection. In fact, many process engineers with whom I meet admit this is the situation today.

Maybe holding the goal of becoming a Type I civilization is too lofty or theoretical to be meaningful and therefore not sufficient to drive practical action. Perhaps, but  I think it does provide some value by shining a different spotlight on a problem in need of greater attention – the control systems running today’s energy plants are too exposed to cyber attack. If you think that it’s possible we will achieve Type I status in the next century, then future generations of these same control systems will manage power we can only imagine right now. It’s about time we learned how to secure them.

How real is the danger? Will we win where other planetary civilizations have seemingly failed?

Barbie Versus the Dam Hack

By David Zahn, CMO and GM of the Cybersecurity Business Unit at PAS

I was shopping on Amazon recently looking for a gift for my daughter. The choices are endless, and it seems everything has an electronic or connected element to it now. While researching dolls, I came across a news item that Barbie can now carry on a conversation with your child. The toy utilizes cloud-based speech recognition technology primarily to accomplish this feat.  All conversations are stored so that parents can access them and Mattel, the makers of Barbie, can use them for product improvements – something typically done in speech recognition initiatives.

The reporting focused on the understandable risks of having potentially sensitive conversational data from your child (and future consumer) in the hands of Mattel as well as on the challenges of securing this data and preventing access to your child from the outside world. Imagine hackers stealing this data and deciding to publish it or holding it for ransom were it sufficiently embarrassing. These are fair concerns – ones that I share and, as a result, I am now looking for another gift to purchase this season.

In a separate news story, the Wall Street Journal had an article on the discovery of a control system at New York’s Bowman Avenue Dam that was compromised by an Iranian hacker group in 2013. The attack was focused on gathering information, so it was deemed of little threat to the community. However, it did raise serious alarms in the government based on the group responsible for the attack. It was particularly notable in that it illustrates how vulnerable our critical infrastructure is to outside attack.

Of the two cybersecurity stories, Barbie received more coverage. So, why the difference? Both are potentially damaging, but only one has the opportunity for catastrophic damage and loss of life (were it another dam). Why is the breach of information more concerning than the breach of process?

There are many answers to this, but I think two are important to highlight. First, information breaches – particularly financial or personal ones – directly impact the individual and are easier to understand for the public. Losing your credit card data, for instance, is more relatable as many have experienced this attack personally. Second, companies are not compelled to disclose process breaches to the public. When they do, they are more difficult to understand in terms of what happened and potential impact. Also, the fact that these breaches have only done damage in a limited number of instances just does not stir public excitement.

Although understandable, is this fair? Clearly the answer is no. Media needs to take greater notice of the potential threats to our infrastructure. Sunlight is the best sanitizer. Bringing more attention to infrastructure risk will accelerate governmental and industry action to secure our systems.

What do you think?  Does the media need to do a better job covering the risks to our nation’s infrastructure?

Why Would You Use an IT Solution to Solve an OT Problem?

IT Solutions for OT Problems
By Nick Cappi, Director, Global Business Development for Integrity Solutions at PAS

While automation technology has been around for decades, protecting control system assets from a set of very modern day threats is relatively new. In the face of growing risk, companies are challenged by continuously evolving best practices and standards. With this new territory, comes confusion (multiple standards, changing regulatory laws, hyped news stories) and often frustration (deciphering vendor claims and managing security for proprietary systems).

Lately, companies have expressed confusion over understanding which cybersecurity solutions reach far enough into their control system assets. The desire to overcome this confusion stems from industry recognizing their most valuable assets are their most vulnerable as well as standards bodies publishing OT-specific compliance requirements (e.g., NIST, Q-CERT, and NERC CIP) during the last few years.

Unfortunately, the definitions of OT and IT – at least as far as the control network is concerned – have become muddied from quite a bit of vendor marketing speak. Let’s see if we can get a clearer definition of terms. First, cybersecurity that strictly focuses on Windows machines, switches, routers, and firewalls is by definition IT cybersecurity. It doesn’t matter if those devices are in the control network or not. OT cybersecurity, which includes the IT systems in a control network, also includes those proprietary, often heterogeneous control systems that sit behind the Windows machines, such as the Honeywell TPS, Yokogawa Centum, and Rockwell Allen Bradley PLC, found in plants today. Think of OT cybersecurity’s purview being a superset of all assets in the control network.

While companies absolutely need IT-based security, by no means does this sufficiently address the control systems at the heart of OT cybersecurity. A solution must account for I/O cards, controllers, control strategies, sequential function charts, registers, rungs, graphics, and other detailed configuration data. Without this information and the ability to analyze it or watch for unauthorized change, cybersecurity risk remains within the control network.

Here’s a quick way to evaluate whether an OT solution is focused on truly OT cyber assets. Ask yourself if your solution addresses these three real-world cybersecurity scenarios:

  1. Can you determine your enterprise’s exposure to a published ICS-CERT vulnerability focused on a specific manufacturer and model of a transmitter?
  2. Can you detect an unauthorized change to a safety system’s logic?
  3. Will you identify the next Stuxnet attack that successfully gets past IT protection layers, such as glued USB ports, A/V software, and firewalls?

If you answer “yes” to all three, then you have an OT cybersecurity solution.  What do you have in your control network?


Is Your House in Order?

By David Zahn, CMO and GM of the Cybersecurity Business Unit at PAS

clipboardA few weeks ago, an inspector visited my home as part of a refinance process. My family is good about home maintenance activities, but he did remind us to caulk and patch the exterior of our house each year. This preventative maintenance best practice protects our most valuable asset from the elements. This visit reminds me of another visit – one I recently made to a large power company. We talked at length about preventative maintenance within their fleet and how they process patches that address identified vulnerabilities. Their process had many holes in it, leaving vulnerabilities unaddressed and unmitigated within their control networks.

Their patch management process comprises four steps in total. First, a central testing group receives patch information from Microsoft and control system vendors. Second, the group captures the information in a Microsoft Excel spreadsheet and passes it along to plant personnel for evaluation. Third, plant personnel, at their own discretion, evaluate the same patch information from Microsoft and control systems vendors. Finally, they then decide to implement the patch, mitigate the risk, or potentially do nothing. The testing group has no visibility into what the plant ultimately does or when they do it. Each plant makes independent decisions that may or may not have consistent outcomes. This is clearly not an ideal, closed-loop process with auditable transparency.

Industry best practices dictate that patches are evaluated immediately matching Microsoft updates and vendor bulletins to an automatically updated cyber asset inventory. A centralized testing group validates applicability and readiness notifying plants of the need to implement the patch or to take steps to mitigate risk. Workflows automate testing, implementation, and mitigation activities providing full visibility into compliance status. Workflows also provide electronic breadcrumbs that aid in internal and regulatory audits.

Applying information technology (IT) standards to operational technology (OT) challenges is a common mistake we see in the industry. The differences between the two are stark and many times unclear to personnel in each camp. Nevertheless, OT can benefit from certain tried-and-true approaches such as workflow automation to drive good patch management practices. Doing so means the “house” is more secure with fewer vulnerabilities exposed to the elements. It just takes a little more preventative maintenance than what most houses (or plants) receive today.

Is your patch management process transparent, automated, and closed loop? If an inspector came to your “home,” what would they find?