Stay Alert: Advanced Security Monitoring & Incident Response for IBM i

Maia Samboy    00:00

Okay, welcome. This is our last webinar in our security series, so if you’ve been with us the whole time, thank you. We’re glad you made it this far. And if you missed any of our previous webinars, I will be sending out those recordings as well as the recording of this one after it is over today. Welcome to Stay Alert Advanced Security Monitoring and Incident Response for IBM I. This session will be about an hour and we’ll try to leave some time for Q and A at the end. We got a lot of really awesome demos today that will be shown by Aaron, our chief architect, and we also have Dan, our CEO here who will be our experts today. We’re really excited to get started. So I’m going to hand it over to you, Aaron.

Aaron Magid     00:45

Okay, great. Well, thank you for the great introduction. I’m excited to get moving here. All right, well, let’s jump in and let’s get started.

Aaron Magid     00:58

Hold on a second here, let me get my screen share going. There we go.

Aaron Magid     01:05

All right, well, welcome everybody to our, our Advanced Security Monitoring and Incident Response for IBMI webinar. It’s good to be back for, for another installment of this webinar series. So today, first thing I wanted to do is go through a little bit about what we’re going to talk about here as we, as we go through this, this webinar. So there’s a couple of main points that I want to get through. One, we’re going to start off, as always, I think, by, by talking about why are we doing this presentation, right? What, what are the key things that, that motivated us to want to talk about this subject? Why is it important? We’ll talk about some core terms around monitoring in the application space and you know what, how that applies to, to the security of our applications. We’re going to talk about some of the blind spots in monitoring traditionally that, that have created an opening for attacks within the IBMi world where we’ve seen, we’ve seen some of those things be exploited in the past. And once we’ve talked about that, hopefully at that point we’ll have a clear understanding of why this is important and what we’re talking about here. And once we’ve got that, we’re going to establish some contrast. We’re going to talk about how we can set up our monitoring so that we can actually address those kinds of threats and so that we can actually see them and mitigate those threats before they become too big of a problem. Once we’ve got that foundation, we’ll go through some Attack scenarios or some things that I want to, to go through with you guys. Some of the things that I’ve seen in the past, some of the things that, that are, some of them are industry recommendations, some of them I, I have personal experience with and I’ll get into some of the details around those. So let’s get started. So first of all, let’s talk about the threat landscape. There is a broad threat landscape, right? I mean, and that’s why I like, I like calling it a threat landscape because it sort of gives you a sense of, sort of the, of what these attacks can comprise on an IBM I environment or on an application. We’re going to be focusing on a very specific aspect of IBM I security that traditionally in my experience working with a lot of IBMi companies has been neglected in the security monitoring and mitigation space. And that’s specifically, we’re going to be talking about application level attacks. So an application level attack is an attack that specifically takes advantage of valid application functions. So I want to contrast this against other types of attacks that we’re generally aware of that people know about, right? We talk about infrastructure attacks, right? Where some, or where, where someone gets in, a malicious actor is able to get into your physical server, your physical hardware, they’re able to sign in, they’re able to get around your application. That’s very common, especially in applications that have things like menu based security, right? Where I have a user profile on the system and I’m able to get under the security layer of the system and just go at the underlying server, go, go at the underlying IBMi and just talk to the database. That’s not what we’re talking about here. We’re talking about application attacks which are attacks that go through your application. And these are where monitoring is really, really important. And sophisticated monitoring is really, really important. And the reason for that is an application layer attack abuses valid application functions. Valid application logic for nefarious purposes. For example, let’s say that I have an API that connects to my IBM I with a service profile. It’s always the same profile, no matter who’s using it, which is a good architecture for a lot of reasons. I’ve got an API that does, that always connects to the same user. Some of my API endpoints act on unprotected data on let’s say, user account information like, you know, whatever names and email addresses. Yeah, I want to protect that, but it’s not, you know, the most critical thing. And then I’ve got API endpoints that interact with Payment information or Social Security numbers or that kind of data that I really, really need to make sure I am especially protective of.

Aaron Magid     06:02

If an application is accessing both of those resources with the same service profile, what you’ll often see, especially if it’s going through like an ODBC connection, what you’ll often see is in IBM I monitoring, the whole thing appears as one flood of the same traffic, right? We’ve got one user coming in over one job that is accessing those different resources. But now let’s say that the access control system in the API fails, meaning a user is able to come in, talk to the API. They’re only supposed to be able to access the unprotected features, but they’re able to get around it and they’re able to access the protected data. What happens on the IBM I is it gets a connection from a known service profile that normally accesses the protected data, accesses the data, sends it back. From the IBMI’s perspective, nothing has happened. And that’s one of the critical pieces here. If all we have on our IBMI monitoring is, for example, exit points on our database connections, right? Which is a wonderful thing to do and very advanced. If all we’re doing is monitoring the IBMI system events, we will not see that kind of action. And that’s where application monitoring comes in. Because those kinds of attacks are dramatically spiking, right? What’s happening is hackers are realizing that this is actually a great way of getting into an app, of getting into a system, right? Instead of trying to get at the underlying infrastructure. If I can take over your API layer, if I can get in through there, because it’s maybe a poorly built API it’s using, it’s not really my main focus. It’s just something that I had to do because my boss told me I had to do it. It’s a very common story I’m going to. Or actually. Or I’ve got a Net application that’s talking to my ibmi. So, meaning the underlying database team is separate from the application team. They don’t really necessarily talk to each other so much, right? That creates a blind spot for me to be able to get in and abuse the application to access things that otherwise I shouldn’t be able to access. And we’ll see some actual examples of that. So with those rises in this type of attack and with the problems in IBM I monitoring systems, ability to detect them, this is why we wanted to focus on this topic today. And I also wanted to make another comment about Monitoring, which is that when I was, when I was in college taking classes on security and working through those, you know, all that stuff that I had to do, I had a really interesting moment in a. In a security course where they brought in an expert and he came in and he said to us, he doesn’t focus on prevention. So he gave this whole. We had this, this whole series of classes with this guy. A very small portion of it was actually focused on prevention. What he focused on, and I’m seeing this as a trend in the industry, I think it makes a lot of sense. What he focused on was early detection and recovery. Meaning what? What he said to us was, when you are trying to prevent attacks from coming in, every attack is a surprise attack. You don’t know where they’re going to come from, right? And you’re building new features, you’re deploying new code, new APIs, new things like that, It’s a very hard thing to make sure that the entire application, everything that you’re doing, is absolutely bulletproof.

Aaron Magid     09:43

So what he focused on is we need to be able to see what’s going on in our system so that the moment something fishy starts to happen, we know about it and we’re able to shut it off. We’re able to deal with that as quickly as possible. That’s a much more stable strategy than trying to make sure up front that every possible attack, every possible endpoint, is absolutely bulletproof. Because it’s just a very difficult thing to do, and it’s a moving target. It doesn’t mean that you shouldn’t try. Right within Aerodynamic Connect, we do a lot of work to make sure that these APIs are bulletproof. And to the credit of that work that we’ve done, we’ve never had somebody actually get through that, that malicious actor actually get through the API layer. But it’s an important point that that alone is not enough. We also need to be watching what’s going on so that as that target moves, we can make sure that we have that extra layer that keeps the system. Keeps the system secure. Now, one of the other pieces that’s important to understand why monitoring is so important is we got to get inside the mind of a hacker for a second, right? Some hackers are disgruntled employees, right, who want to damage the company. But most attacks aren’t about that, right? Most attacks are about money. Most hackers are coming in. They want to either siphon off your data so that they can sell it or abuse it if it’s, you know, for example, payment information, or they want to drop ransomware on your machine and lock it up or steal your data, hold it for ransom, right? It’s about, it’s about getting access to the system. So for a separate goal, right? Not just to take down the system, it’s. The goal is to actually do something with it or use it as a stepping stone for an attack to somebody else. And the important point about that is that that means that system takedowns are not usually in the best interest of a hacker. And if I’m a hacker and I realize that I have access to one of your systems, the last thing I want to do is put some software on there that very loudly makes my presence known, right? By, for example, bringing down your system in a way that doesn’t have a big payoff for me, right? Because the moment I attack the system, the moment I actually bring it down, you’re going to know that something’s on there. The much better thing for me to do is to sit there on your system for 20 years, siphoning off your data silently, right? That’s the much more common attack strategy because that gets me something that, you know, I can be continuously selling or abusing your data or abusing your system for a very long time if you don’t notice that I’m there. And that’s why we, why again, why we want to make sure our monitoring is really solid, because we have to be able to detect those things. We have to be able to attack that malicious use and even if it’s not directly impacting the company. And one interesting thing that, that I’ve, that I’ve learned over the last couple of years in studying these attacks is that oftentimes is denial of service attacks or system takedowns or outages, things like that. This principle is so important and is so universal to attacks that oftentimes the attacks that are system takedowns like denial of service are actually just smoke screens for what’s actually happening, which is data theft, right? So usually that’s what’s happening here. So again, if we’re watching what our applications are doing, that’s going to help us close off these kinds of exploits that may or may not be happening without us even knowing about it. And again, keep in mind that what we’re talking about is valid uses of applications or malicious uses of valid application functions. And this makes it a difficult thing to, to pin down. So let’s talk about monitoring. So I’ve talked about monitoring, I’ve talked about application monitoring, and that we want to see something, let’s define that a little bit more. So there are, when we talk about monitoring, there are generally two major categories that we want to talk about. One is development time monitoring, that is watching our code, watching our processes. We talked a about that in prior webinars. But there’s also the runtime monitoring, which is watching what our system is doing when it’s actually operating. We’re going to be focused on runtime monitoring today because generally speaking, development exploits are, are not going to be a way where we’re going to be stealing data actively. Right? That’s where you, the development monitoring helps you stop the malicious code from getting it in the first place. But the runtime monitoring is what we’re interested in if we’re actually detecting attacks when they happen. So that’s, we’re going to be focusing on today. And that’s where we get into dashboarding systems. Right? But these systems that we look at are more than just dashboards, right? So for example, if I bring up something like Grafana or Logly or Splunk or New Relic, Wrap, Dynamics or Datadog, probably spend the rest of this webinar just listing different platform names. When we get those kinds of systems, we need to be able to bring in our data and visualize it. But these applications also have critical features that go beyond that. These applications often have features around alerting, which is obviously important for getting proactive alerts about conditions that happen in the system. But there’s also actually a very powerful type of, of tool that most of these platforms are developing now, which is, which is analysis toolkits, which often include machine learning capabilities. And those are toolkits that will help us sift through that massive flood of information that’s coming through our monitoring systems and actually pull out the useful endpoints. And what’s funny or what’s actually what’s powerful here is that when we want to, when we want to pull out that data, right? There’s actually an example of this on this screen in the sample dashboard. A great way to look at this is this kind of system like this bottom left widget that I have on the screen where I’m able to look at the different requests that are coming into the system. Right? That’s sort of a base level that we want in our system because that will help us see if there are spikes in traffic to certain endpoints that may indicate a, an attack condition. And we’re going to, we’re going to get into that, we’re going to get into that more as we move forward through this. But I just want to point out these error statistics and these call statistics that are going to be very important for our monitoring as we move forward.

Aaron Magid     16:50

So first of all, before we jump in any farther, right, we’ve talked about why we need to be monitoring and we’ve talked about what we mean when we say monitoring. Let’s, let’s just jump in and show what this looks like. When I actually need to monitor a system and pull critical data out of the application, what does that look like?

Aaron Magid     17:18

So what I’m going to do here is I’m going to pull up my, my 5250. I’m going to use something that we’ve actually used elsewhere in our, in our webinar so far. And the point that I want to drive home here is that I can get my IBMi application in here and I can actually extract meaningful data from my IBM I application. Right, that’s. And again, that’s, that’s one of those baseline capabilities that we need in order to do application monitoring. We have to get our application hooked up to our monitoring system. Right? Our application has to have the ability to send data about what it’s doing to the monitoring system. That doesn’t mean our application is scanning for attacks. And notifying somebody of attacks is a critical difference. We’re not necessarily getting the application set up to just to, to actually do the security scanning. What we want the application to do is to tell us what it is doing at all times. And then we will leave it to the monitoring platform to handle the analysis of those events and of those functions so that we can extract the information that we need out of the monitoring platform. So I’ll give you an example. This is an application that you guys that we’ve seen if you’ve, if you’ve attended any of our previous webinars.

Aaron Magid     18:39

So I’m gonna go in here and do my, my MFA call here. I’ll just pull up my phone here and just approve that. Let’s go ahead and do that. And that’s going to let me into my system. And then while we’re waiting on this, I’m going to go in and I’m going to do another one.

Aaron Magid     19:00

And this time when that notification pops up on my phone, I’m going to go deny. Now, the key point here is that as those calls went in, you notice they showed up in my dashboard. So I want to focus on this for a second. Explain what happened here. I have an IBMI application. I have an exit point that runs. I have a program that runs when I connect to my IBMI when I open a 5250 session. It is written in IBM I languages. It is an IBM I program that handles my multi factor authentication. But the critical piece here that I wanted to point out is that that application is then talking to my dashboard and saying, hey, I got these authorization requests. Now, critically, it’s not just alerting on the case where the multi factor authentication was denied, right? That’s definitely a case where we want to alert on, right? Somebody there’s a login attempt, somebody gets an MFA prompt and they deny the login. That’s definitely something we want to alert on. But what might be less, less intuitive is we also want to be alerting on when they allow the auth, because we need to be able to establish the patterns and watch what’s happening in our system, right? So this will allow us, this allowed versus denied will allow us to establish trends in what’s happening in the system. Getting the data out of our application will help us with this, right? And just to give you an example of this, if I see this failure and then I saw the success, this is actually a case that may help me to realize that this is not actually a malicious case. It actually may just be a case where a user hit the wrong button, right? So there are, there are pieces here where we want to be watching these. We want to be watching all of the information about the application, not just the, not just scanning for specifically error cases. And that gets into some of the details about application monitoring in our, in our systems. Right? I just want to really drive home this point that application monitoring is, is distinct from what we do with traditional IBM monitoring platforms. I Application monitoring is, for example, not system metrics, right? We’re not necessarily looking at disk utilization or CPU usage. We’re not looking, we’re not even necessarily looking at exit points on our database access because that’s more at the database level, that’s more at the IBMi level. What we’re looking at is specific events more abstract than what we may be tracking in our other applications. So we’re looking at specific application and business specific events that are happening in our systems. And then we’re trying to establish trends and understand what’s happening in the system by sifting out the relevant information from that, from that data. That’ll include error events, that will include timings, that will include business metrics, that will include success events, that will include informational messages. It’s going to include, hopefully A huge amount of information about what’s happening in our application. And we need that so that we can have our tools establish trends. It’s actually an interesting point here that with application monitoring we actually want to have a huge amount of noisy data in here because we will have tools that will go in and actually work with that and find the trends for us. We’ll be able to query that information and find it, but we can’t query stuff that we don’t have. So we actually want to have a huge amount of data coming into the system.

Aaron Magid     22:54

And when we have all of that information, when we have all of that data available in our applications, that will allow us to establish baselines in our application and it will allow us to detect anomalies. Right? A great example of this is when we look at. Actually I. I like to focus on. On database connection hijacking or SQL injection is a very common way of doing that, right? Which is when a hacker is able to take advantage of an existing database connection to run queries that the application was not coded to run, right? Basically supplying their own code. It’s a form of code injection, right. It’s basically an ability for the, the attacker to run code that they want. And especially if an attacker is siphoning off data, the way that that will present is more data than usual is being pulled out of a particular file or data that is not normally pulled is being pulled. And there’s a key point there in the way that I’m talking about that, that I want to focus on, which is, I’m saying more data than normal, right? Different data than normal. Part of the question is what’s normal, right. And if you work with application monitoring, one of the first things that you’re going to do in these applications is you’re going to establish baselines. You’re going to have tools out there that are going to read all of this data that is coming through your system and establish those baseline trends that can then be used to compare against future performance or future events so that anomalies can be detected. And that’s what our monitoring tools are going to do for us. They’re going to allow us to find those anomalies. And there’s some pretty powerful features of. Of these application platforms. One of my favorites, aside from. Well, one of my favorites is the machine learning toolkits, which basically are provided. There’s a toolkit for basically every platform, which is to give an example, right. Splunk, if anyone’s using that, you have a machine learning toolkit available in there. The Elastic Stack has machine learning toolkit. And basically what they’re doing with those is they’re saying they’re helping you configure these ML systems, machine learning systems that will go through your data and determine what normal means for you so that then when a subtle anomaly happens, they’re able to detect it at a slightly simpler level. Not slightly, at a substantially simpler level. There are also tools that will do in, in all of these platforms that will do analysis based on standard deviations from number, from metrics that you’re tracking and things like that that will help you to do it without the use of, of heftier machine learning systems. But, but again, there are really powerful capabilities of these systems. One of the, one of the things that, that that kind of tool can help with is probably my favorite hack. You know, of all the different attack types that I look at, one of the coolest that I’ve seen is a timing attack. And that’s one where detecting anomalies is really important because what a timing attack does give you an example of this, a timing attack. Let’s say that we’re trying to crack a password. And a common way of checking a user’s password. Let’s say you’ve got a password that’s in a character string, password is going in. Let’s say it’s, you know, 20 characters long. And let’s say that the way that, that, that what I’m doing in my code is the password comes in and what my application does is it says is the password equal to the correct password for this user? Right. Meaning I load the username and then I check. Do the passwords match? It’s fairly common algorithm for, for checking a password. The thing is, under the hood, what the system, what your programming language is doing is it’s probably checking the length of the two values. If they’re not the same, immediately saying it’s wrong. If they are the same, it’s going through each character of the password and checking and as soon as it finds one that is not a match, it’s going to stop checking and say and return an error. And the critical piece here, this is why this is probably my favorite attack. The critical piece here is that if I have the correct password, it will take longer for the system to check it than if I don’t have the correct password. And the more correct my password is, the longer it will take for the system to check it. So what a timing attack does is it tries out passwords and measures the amount of time that it takes for the system to check the password and how long that response happens. And then using a statistical analysis figures out whether the past, the new attempted password is more correct or less correct. And it allows systems to break passwords within a couple hundred or a couple thousand attempts, rather than a couple trillion or quadrillion attempts. It is a phenomenal tool for hackers when this type of vulnerability is present in a system. And it’s a really easy vulnerability to have in a system. Now why am I talking about that? One of the reasons is because I think it’s really cool. But the other piece is the way to detect that when someone is doing that is to be watching those authentication attempts and looking for anomalous traffic. Right. In order to detect that, what we have to be able to do is see that there is a spike in login attempts, right? We have to be able to watch the rates at which things are happening and see that something is different than normal. Right? And we have to be able to sift that out from the data that we’re getting. So that’s where where this monitoring is, is again, gets really important because we have to be able to do complex analysis of relatively subtle trends in our application, right? Because again, if I’m an attacker, I’m not necessarily going to run all thousand attempts all at the same time, because I don’t want to trip your alarms. I’m going to run them at a slow trickle for a couple of days, right? So we need to be able to detect subtle patterns in our systems in order to be able to counter those types of attacks. Another key piece of this monitoring is that it allows us to put all of our application data from all of our applications into the same platform. Right? And I’m only going to spend a second on that. That’s just important because there may be attacks. In fact, they’re a very common strategy in attacks is, is to do what’s called lateral movement, meaning I get into one server on your app, on your network, and and then from there I jump onto different parts of your network, different parts of your application. If I can’t trace across applications, it’s going to be hard to detect the trends that notify me of that type of attack. Right? Because if I just have monitoring of my IBMI and I have separate monitoring of my. Net applications, then when events start happening in both of them, I won’t actually know that the same person will not know that both of those things are happening. Right? Which means that we don’t get the full Context and we don’t actually know what’s, what’s going on in our system. So all that together is to say we need to get our IBMI applications into our application monitoring systems and we need to be able to sift through all of that data that’s coming through so that we can detect anomalies and we can establish trends across our entire infrastructure. And that is what application monitoring is all about.

Dan Magid         31:12

Now, if you, if you know that one of my favorite hacks was probably one of the most dangerous hacks ever, which was the XZ utils attack that, where, where hackers over a period of years were able to insert into this, this compression utility was the most popular compression utility in Linux malicious code that basically compromised ssh. That opened up SSH and they could get in and it was out there nobody knew about. The only way it was discovered is a Microsoft engineer happened to notice a 500 millisecond delay in a process he was running. And he kept running it over and over again and could not figure out where was that 500 millisecond delay coming from. And he was able to track down that there was this malicious code running inside of this, this utility and that the danger of that hack was it basically opened up almost every Linux system to, to these hackers.

Aaron Magid     32:10

Right, right. And that’s a critical point because what he did, I don’t actually know the tool that that engineer used, but these types of monitoring systems would show you that. Right? They’ll show you, hey, something’s happening, it’s taking too long, it’s slow. Right. And from there, that’s, this is how we get that data. Right. It’s not usually a developer just looking at it, you know, with a stopwatch and seeing that something’s taking too long. Right. It comes from aggregate metrics of these applications, which again, we get out of these types of monitoring dashboards. Yeah, that is, it’s, it’s, it’s a little bit, I always feel a little odd saying the phrase my favorite hack.

Aaron Magid     32:53

But, but, but it is, you know, hackers are smart, at least some of them, and unfortunately the smart ones are really, really dangerous. So, you know, we, again, we need to be making sure that we’re, that we’re watching this data. Thanks, Dan, for that, that, that example, I think that’s, that’s great. So looking at our metrics and our, our monitoring data, there are three types of metrics that we would look at in an application here. And if you’ve been to any of my monitoring presentations, you will have slide that. At a very high level we need to be able to see high level application metrics. Things like how long are our calls taking, how many calls are we getting, what are the error rates, that kind of thing. At a, at a more granular level we need to be able to see log data from our applications that we can see what’s happening in the system and we also need to be able to, to compare that against what’s happening in the other parts of our system. Right. And that’s where telemetry and trace data comes in which allows us to see what’s going on as data flows through our applications. Right. So these three together metrics, logs and telemetry allow us to see the entire picture of our application and allow us to notify our, our security teams or our on call engineers, whoever needs to know about it when something out of the ordinary is happening. And again critically it allows us to actually understand what ordinary is. Right. And I think that’s, that’s one of, one of the conversations that I get into a lot when I do monitoring implementations for IBMI companies is what’s, what’s normal? You know, we, we build an API and one of the things we want to watch is is it taking too long? Well, what’s too long? That’s one of the discussions that we often have. You know, what is reasonable for how long extremely complex system like this should take. Right. So those are the, that, that’s another piece where these tools are, are important is just establishing the baseline. Now quick thing here because I always want to give a, I, I always try to focus on useful information in here, actionable next steps. So I wanted to point out here that my recommendation to everybody who’s here is that if you aren’t already doing monitoring of your IBMY application, check with your companies to see what tools they already have in the organization. Most of you will find if you don’t already know that your application has one of these tools available, they’ll have a loggly or a Splunk or Grafana or new Relic or something like that, AppDynamics Datadog. They’ll have something in there most likely already that other teams are using and putting their data into. So my recommendation for everybody who’s here is if you haven’t done this already, after this webinar, talk to your team, talk to the other teams in your company, see what they’re using because your IBM I can play in that environment despite what you know, whatever they might think, but they probably don’t think it can, it can, we can get the metrics in there and, and we can help you do that. If you, if you need help with it at Erdani, we can help you. We do it a lot. And the other piece that I want to mention here is if your company does not already have a standard, we offer a prepackaged Grafana setup. Grafana is a dashboarding system that can do all the things that we’ve talked about here and all you need is a Linux or a Windows server to drop it on and we can get that dashboard on there for you. It is an open source tool, so it doesn’t come with any fees to Grafana or anything like that for your company. You just drop it on your server and our installer will unpack it and start reading data from your ivm. So just a key point here that we can get, you can get into this very quickly. And, and again, if you need any assistance, let us know and we can help you out.

Aaron Magid     37:11

All right, so with that all in mind, let’s just take a look again at what we saw at the beginning here, our monitoring. When I ran my outbound call here, right, to get my MFA alerts into my dashboard, my program sent some data over to my dashboard here. Now this dashboard, what I wanted to point out is this dashboard is Grafana. So I am actually using the tool that we’ve just been talking about, right? And again, this application is receiving data from my IBM I and visualizing it for me in a way that I can see, right? So for here, for example, I can see the number of calls per minute that are coming into the system. I can see how many calls I’ve completed. I can see error rates, I can see alerts. I’ve got it set to the last five minutes, so the ones from before have been cleared, but I could get those again if I want to change the time range, right? I’ve got access to all of this data, processing time, payload sides, etc in my application. So these are again very simple to put up once you have Grafana. So again, let us know if, if you need help setting this up. And again, the same data that the application is exporting into Grafana can be exported into any other application or any other standard monitoring platform. Okay, so I want to transition here into, into some of our what, what we actually do about these issues. In order to do that, I want to review some of these blind spots that we’ve talked about, right? The IBM I traditionally has monitoring systems, right? Most of you probably have IBM I monitoring Products installed, you have something from vendors that is, you know, that’s taking care of your monitoring, that’s watching for security events, login events, database access, those kinds of things. But one of the major blind spots that we typically have is application visibility. Right? We need to be able to see what’s going on in our applications. And also we often have very limited log analysis. What I mean by log analysis, I mean getting that flood of data and actually extracting meaningful trends from it. There are of course vendor products that will do that kind of thing with the IBMi history logging and with job logs. But those tools, in my experience, do not come anywhere close to the capabilities of these industry standard monitoring platforms. And so one of the key points that we want is we want to be able to get into those, those baseline tools, establishment tools, the, the machine learning toolkits, and those analysis tools that will give us really powerful custom visualizations of our systems, which requires these, these other dashboarding systems that we’ve talked about. Critically, the IBM is separate from the rest of the organization usually in terms of its monitoring. If you’re using an IBMi vendor’s product for monitoring, you’re probably not getting your Windows applications in there too. And again, that means that we can’t correlate between the logs except manually. Right? The only way to notice trends would be to actually be looking at both of them, which again, when you have large flows of log data is essentially impossible for humans to do. Right? And that results in a lot of missed security threats. Actually one of the best examples that I can give is when we go live with APIs with our customers, almost everybody when they go live will shortly see a massive flood of weird looking requests come into their API that can come from one of two sources. Either it’s from hackers trying to scan this brand new API to see if it has any known vulnerabilities. Like for example, they’ll hit, you know, like endpoints that, that are known vulnerabilities in old WordPress applications or mail servers. They’ll try to exploit log4j, you know, the vulnerabilities and things like that, even though that’s not possible here, right. They’ll try to do a whole bunch of exploits all at once, see what they can find. Alternatively, it may also be done by a securities tool owned by your company, which is attempting to basically do the same thing that hackers do so that it can notify you. But either way, the interesting point is that most, most of the time when that happens, it’s a surprise It’s a new thing for the IBMI team and the critical piece here is that it’s new to the team from a monitoring perspective, but it’s not actually new in terms of the traffic that’s happening to their system. Their system is being scanned periodically. But the fact that it’s a, that it’s a surprise tells us that people don’t know that that’s happening. And so that’s where I say this is where we have missed threats, right? We’re not aware of those kinds of things happening up until we set up the monitoring. Then we say what’s going on here, right? What’s happening? Right. So we need to make sure that, that, that we’re getting this data from more than just our API layer because that, that’s, that’s been one of the, one of the things that I, the concerning things that I’ve seen around the community.

Aaron Magid     42:17

And again, remember what we’re seeing in the ecosystem is that these types of attacks are increasing, right? So we want to make sure that we’re, that we’re effectively defending against them. So when we work with our monitoring platforms, we’re going to get the data in, they’re going to centralize it for us, they’re going to learn from it and critically, they’re also going to alert and they’re also going to establish, they’re going to establish time based data for us that can help us see what’s happening over time. Again, especially when you’re dealing with a sophisticated attack, things are often happening in stages, they’re happening in different places and they’re, they’re going to be a little bit, there’s going to be a lot of information from a lot of different places that can help you find when those kinds of things happen. So I want to make sure that we get these into our platform so that we can use it and we can access them. So a question here, right? How do we actually do this? We’ve talked about connecting into our monitoring systems, we’ve talked about why we want to do that, but how do we actually get the data in there? Well, the simplest thing to do is, is actually to use tools that are able to talk to metrics, applications to monitoring systems. Most of these monitoring systems have open source toolkits. So what we need to be able to do is we need to be able to get our monitoring data from our IBM application into our open source toolkits so that they’re able to get it to the dashboarding systems. And, and that’s, that’s One of the things that we’ve put a lot of work into within aerodyne Connect. We’ve, we’ve spent a lot of time getting those applications ready to connect our IBM applications to the dashboarding systems. And again, one of the things that, that allows us to do is aerodyne Connect is already connected to the dashboarding systems. It, it itself knows how to do this. It knows how to talk to every monitoring system out there. And so what we can do is using those same interfaces that we use for making calls to APIs, using the same interfaces that we use for making calls to Kafka or for doing EDI processing or security processing, whatever else we’re doing, all of those, those interfaces we can use to extract metrics from our IBMI programs and we can allow our IBMI programs to, to send events, send log data, and send metrics over to our aerodynamic connect system and that can then act as the bridge to our dashboarding systems. Again, just following its general use case of connecting everything to everything, right? We’re using this in another way to say I can get my IV my data into a dashboarding system and from there it’s a pretty straightforward piece. All we need to do is take our IBMi programs and have them send character data that has the metrics in it and the application will be able to send them up. So this is really the key piece here. Without a sophisticated layer like that, there’s a couple of somewhat more difficult ways of doing this. The dashboarding Systems do have APIs, but again, we need to make sure that our system is really streamlined. So that’s why we put a lot of work into making sure that this is possible.

Aaron Magid     45:39

One of the things that I want to point out here also about our monitoring is that in addition to my IBM I monitoring, I also have in here my API system and its monitoring. Right? So I can see what’s going on with my APIs, which are my aerodynamic connect applications, and also with my IBM application. Right. And again, that aggregated data is what’s going to allow me to establish trends in my system over time so that I can see if something is happening across my application that I can more readily detect problems. So hopefully at this point I’ve made it clear why we want to do this and, and what it looks like and how we can do it. I wanted to go through some, some attack scenarios, just some things that I’ve seen in the past that, that I think are, are, are powerful and, and I just want to do this to give you A sense of how to think about these application level attacks and some of the things that are possible, right, These are different from what an attacker may try to do when they’re trying to get into the underlying physical or the underlying IBMi application or, sorry, IBMi server, right? They’re not necessarily trying to get 5250 access in a user profile or ODBC access and user profile. They’re trying to, again, abuse application functions for their nefarious goals, right? And again, the reason why a hacker wants to do that and operate in that way is that to the untrained eye, if they’re doing that, it all just looks like normal application traffic. So if you’re trying to subtly siphon off data, it’s a great way to do it because if you’re, if you’re not really being sophisticated about your monitoring, you’re not going to notice when those things happen and they can go on for a very long time.

Aaron Magid     47:36

So one of the key pieces here is login patterns. Now I’m focusing on the broader concept of login patterns. This may be application logins, it may be these, like these MFA authorization that we’re looking at. But one example here, one very common, one very common pattern to look out for is again, unusual login patterns. And what the, what I mean by that is when you start to see, for example, repeated logins for a particular user, when you start to see logins for a user that, that are coming from a source, that is abnormal, right? Those are the kinds of things that we want to look at. Or a large number of password failures, right? Those are the kinds of things that we want to look out for in our applications so that we can see what’s, so that we can identify when it might be not a legitimate call. And again, this is one of those cases where we’re sifting through a lot of data, right? If we’ve got, you know, a hosted application where we’ve got hundreds or thousands of users and people are coming onto the system all day, right? Seeking out the anomaly is, is very complicated. And so again, that’s where we want to use these kinds of platforms that can do the analysis for us, right? So just to give an example of this, in my Grafana Dashboard, we’ve seen this. I have on here a widget here that will show me every single one of these authorizations that’s happening in my system, right? And from that I can see if I, even If I have 100,000 of these entries or a million entries in this list, I can still sift out this denial that’s happening here, or I can still sift out all the entries that have to do with this particular user and notice that there are a large number for that particular user which might indicate that there’s a, an attacker trying to get into that profile, right? So that again, getting that data into the platform allows us to search it and query it for that type of data.

Aaron Magid     49:43

Another piece that’s important to watch out for is sudden spikes in traffic, right? This is the kind of thing that can be totally normal traffic or totally valid functions, but is often used to produce denial of service attacks, which again, are often smoke screens for other malicious activities on the system. To give you an example of this, if I have an API endpoint where I’m normally taking in a certain number of calls, and then all of a sudden I see in my dashboard that things are really cranking up, it can be a very powerful thing to be able to see that data and actually pull it out of my dashboard. I want to give an example of this

Aaron Magid     50:30

because I actually have, in my dashboard here, I actually have a, a system that is showing my API calls. And if you notice here what I did, I have a tool that is running repeated calls here. It’s just running it, you know, running API calls in a loop since I started this up. And if we look at the dashboard, what we can see is that all of the rates around my application are climbing, right? Everything around the application is going up dramatically. And again, that’s, that’s an important point because I can see what’s happening on here. For example, I can see that this piece just crossed over a threshold that I set in my monitoring platform, right? So in my dashboard, I can see that I went over my threshold and I can see that these numbers are still climbing, right? So this is the kind of condition that I may want to alert on. Now this particular API, what it’s doing actually is processing EDI data. And this is where I get actually an even more interesting view, where I can see here that I have processed all of a sudden tens of thousands of transactions, and I’m receiving, I was before receiving, according to this, this dashboard, I was receiving 0.418 calls per minute to my EDI integration here before. And now in the last measurement, I received 577. Now I’m up to 615 per minute, right? So I have a spike of over a thousand times more calls that are coming into my system, right? So this is a great scenario for, for alerting on what’s happening in the application. And Grafana actually supports that type of alert. Using our Grafana system, we are able to actually see that data. We’re actually able to set up alert rules in our application here. Just jumped over to my Grafana and I can see over here I actually have an alert that says it’s firing. It says too many requests. So I actually set a Metro or a, an alert in here that is firing now that is sending a message to the appropriate person to say, hey, this is not a good situation. And where it went, it went over to my slack here because this is where I chose to put. It went over my slack and it says, it sent me a message and it says, hey, you’re getting too many requests. Right? At the last sample when the, when the alert was fired, I was getting 384, 384 calls per minute, right? So I can get this data and I can actually see that this is happening, that somebody is trying to get into my system. I can see that there’s a problem in the data flows. I can see that something is going too fast. It’s not necessarily, you know, in prior monitoring presentations we’ve talked about this as what if you have a, a user who writes some bugged code and it gets stuck in a loop, right? And then they start sending too many requests. This, we’re talking about malicious operations. We’re talking about somebody actually trying to take down my system. Because if I was sending, you know, 10 large EDI documents per second to a typical EDI system, it would be a massive tax on the system. It doesn’t take a lot of EDI to, to do a denial of service attack because the integrations are so big. So that’s a, just another key piece to have here. I wanted to talk about one other scenario here which again is one of my, one of my favorites and one of the most important to be watching for. And that is what I’m going to.

Maia Samboy    54:22

Five minute warning also.

Aaron Magid     54:23

Yeah, thank you. This is what I’m going to term, we’re going to talk about here SQL connection hijacking. We’re going to talk specifically about SQL Injection. For anyone who’s not familiar, SQL Injection is an attack type wherein an application is, is running SQL queries against the database and it is not appropriately cleaning input data before it goes to the ibmi. And what that allows attackers to do is it allows them to put, potentially instead of putting a value into a query, it allows them to put SQL code into their query that modifies what the query is doing and what it is returning. And that is, again, it’s one of the nastiest vulnerabilities out there. It is a form of code injection, basically allowing attackers to seize control of the system and do basically whatever they want. But the thing is, we usually demonstrate this kind of thing with saying, what if an attacker was able to delete your entire database? But again, as a hacker, that’s not the main thing I want to do. Because if I delete your database, doesn’t really give me much value, right? It tells you that I’m there and you’re just going to restore it from a backup. Right. What I really want to do as an attacker is I want to retrieve data that I’m not supposed to have. That’s what I really want to do with my system. And I want to show you an example of this. I have here an API that I’ve set up in this application

Aaron Magid     55:54

where basically I take in account data and using that account data, I am using that account data, I am going out to the database using an account id, I’m going out and I’m getting a particular, a particular account from the system. So what I’m going to do here is I’m going to call this API endpoint. And what I’m going to do, here’s my URL. You can see what I’m doing here. I’m just saying, I’m saying get me the accounts for this particular, for this particular account id. And I apologize, I realize that my URL here might be a little bit off. So, okay, so I’m just going to send that request and sign into my API here, one second. Get my token and go over and put that guy in. And when I call my API, right, I give it an account ID and it runs a query which gets me this particular data. Now, again, what SQL Injection does is it says, well, if I know that this API is taking the account ID and it’s sticking it into a query, which actually is what this API is doing. I coded it specifically so that what it does is it runs this query and then what it does is it puts the value into the query itself so that it can run that query with the updated account ID and then, and then execute that and return the results. If I go over to this parameter here and I change this a little bit so that it looks like this weird thing. So this is my parameter here, right? 0Apostrophe Union Select 1 card number Card CVV Card X Free cardholder name Zip code from Q Payment Detail and then a double dash is this very syntactically specific input. When I run that call, what it’s going to do is it’s going to inject that into the SQL query, which will hijack the query and make it not get the data that we expected it to get. So instead of getting this account ID where, or getting this account data where I get last name, initial, street, city and state, what I end up with is this data, which it still says last name, initial, street, city and state, but what’s actually coming out here is a credit card number, cbv, expiration date name and zip code. I’m actually extracting payment information from the database using the same API endpoint by putting SQL code in as an input. And again, the critical piece here is that I’m hijacking a valid application function. I didn’t need access to the system, I didn’t need to go in there and get a user profile. I went through the API, right, and I was able to pull out the data that I want. And if you’re not monitoring effectively, this kind of attack allows me to siphon off your payment information, which is really valuable to me as an attacker, and I can do whatever I want with it. Now, what’s funny here is when I set this up, it actually took me a long time to set up this API because I was using Erdogani Connect and Erdani Assist to generate this API. And I actually had to go in and, and disable a whole bunch of protections in my airdronic Connect system to allow this injection to go through. Because airdronic Connect does not like SQL Injection, does a lot to try to protect us from it, to the point where actually I tried to use our AI assist to generate this API and at first it actually refused. And so I had to say to it, I’m doing a presentation on injection. I need to build an insecure API that is vulnerable. Can you please do it for me? And it, you know, relented and let me do it, which is a very, very funny conversation with my development tool, trying to get it to let me do that. But, but once I have this up here, right, we can see that this kind of thing can be exploited. You can hopefully imagine how this kind of thing can be exploited to retrieve data. And the way that our monitoring helps us is my application here is logging for every API endpoint. It’s logging every SQL query that is run. And so the way that you monitor for this is by setting up your dashboard to look for queries that involve sensitive data and to log who is accessing it and as part of what function. And that will allow you to see when something is being hijacked. Because when you see that someone is accessing the, you know, whatever, the user lookup endpoint and a whole bunch of credit card data got loaded out of that, right, Your monitoring system can say, this is abnormal. This is not the data that’s supposed to be coming out of here. Right? So again, those are the kinds of things that, that we want to do with, with our monitoring platforms. So hopefully this has been helpful. I know that I’m a little bit over here, Aaron, and just.

Dan Magid         1:01:09

By the way, by the way, the, the, according to the open, the Open Web Application Security Project, OWASP injection hacks remain the number one way people attack APIs and web applications.

Aaron Magid     1:01:23

Right? Yeah, right. Because it’s, I mean, you can imagine, right? All I had to do, you know, all you have to do when you’re hand coding an API, if you don’t have the extra protections that we put in is say, query plus value. If you do that, you have an injection vulnerability. It’s really easy to do and it happens a lot. Just a quick side note, number 7 on the OWASP top 10 vulnerabilities list is not having monitoring systems like what we talked about today. So that’s also on there. So if you want to look at the Osprey recommendations, they tell you not to do this and they tell you not to not to not listen to us. Basically they tell you to listen to what we talked about here. So anyway, just since you brought that up. Anyway, thank you all for listening. That’s, that’s what I got for you today. I hope it was useful.

Maia Samboy    1:02:11

Thank you everyone for joining us. Like I said, keep an eye out for this recording as well as the recordings of our other sessions. And if you have any questions or want to learn more about how Eridani can help, feel free to email me mayaerodani or infoerodani and hopefully we will see you at the next one. So thanks everybody.

Aaron Magid     1:02:29

Thank you all. Good one.