Be Afraid: IBM i Security Vulnerabilities in Today's Connected World.

Dan Magid 00:00

All right, the topic of this makes me chuckle a little bit. Be afraid IBMI security vulnerabilities in today’s connected world. So I’m going to start out by talking about reasons why maybe you should be a little bit afraid. In the IBM I world, we have often relied on security through obscurity just hoping that nobody really knows about the IBMI and therefore will not attack it. But what we are finding that more and more that’s not true. We’ve actually had had several customers reach out to us who have, who have had problems in their. IBM is being either under attack or getting attacked. And so we want to talk a little bit about what are the things you can do to protect yourself. So I’m going to start out with talking a little bit about, you know, why be afraid, what are some of the things out there. But my presumption is you already know that or you wouldn’t be in this webinar. So. So you probably are fairly familiar with that. So I won’t spend a whole lot of time on that. We’ll talk a little bit about some of the specific vulnerabilities IBM I that we’ve been hearing about from IBM I customers, and then we’ll talk about what people currently are doing for authentication and some of the issues that expose you to potential security problems. And then we’ll talk about some of the solutions, some of the things that you can do in order to remediate those things and to protect yourself. And I’ll talk a little bit at the end about some of the things that Eridani can do to help you out. As we go through the webinar, if you have questions, you can post them to the chat. We’ll be monitoring that. We’ll try to answer them as we go, or if we can’t, we’ll get to them at the end. Okay, so just a little bit about why be afraid. This is just a current chart of the cost of cybercrime, of attacks on computer systems. And these are in the trillions of dollars. So you can see it’s significant amounts of money and it just keeps going up and up and up and up in the number of attacks keep rising. If you’re involved in, as many of our customers I know are, you’re involved in manufacturing, distribution, you’re somehow attached to a, a global supply chain, those attacks are going up. And the scary part of that is, is that the hackers are actually going through the supply chain and attacking people through the connections they’re connected to. So they’re actually being able to leverage the connections that you have with your partners in order to attack different company systems. In the finance industry. The cost, the individual costs of attacks are rising rapidly. And you can see down here that the number one, the number one attack vector is phishing. The second one is compromised credentials. The scary thing about that is that we’ve often thought that in the IBM I world if you just don’t connect your IBM to the Internet, you’re relatively safe. The problem is, is that so many attacks these days are coming because somebody’s on your network has had their, their credentials compromised. And so even if somebody, somebody, even if you protected your IBM I from the Internet, there could be somebody attacking from inside your network, from some other machine that might be on the network. And the other scary thing about, about these attacks is a lot of them are not the ransomware attacks that you hear about where you instantly know that you’ve been attacked because everything, all your data’s getting encrypted. But there are attacks that are just quietly sitting on your system. Things that are happening in your system that you don’t even know about. So example of that would be like the breach at Target several years ago where they actually got some software onto their POS systems and it was just for months running there and gathering credit card information and user information and, and sending it out to the hacker. So a lot of times these breaches are things you don’t even know about. They’re just happening on your system. And we’ll talk a little bit about that problem. And these are some of the things that our IBM I customers have identified to us as some of the things that they are concerned about as far as the kinds of vulnerabilities they have. You know, one of the very common things we see in the IBM I world are liberal grants of high levels of authority. You know, all the programmers have all object authority or SEC offer authority on the system. So, so I now have a bunch of credentials that have very, very high levels of, of authority. We, we don’t have usually a lot of the password rules that you see in a lot of other systems where passwords have to be changed on a regular basis and there are very strict rules as to what those passwords have to look like. We still have IBM I users who are using menu based security. So if you can get underneath that, you get into the system through an SSH connection or some other kind of connection, you bypass all of the menu based security and you can get at the IBM my system we are also seeing customers who have API endpoints that are just using HTTP and not HTTPs. A lot of customers think that, well, it’s inside my network, I don’t need to use HTTPs. But now you have that insecure communication. We’ll talk a little bit more about that. A really common thing we see is generic ODBC access to data, where you’ve got some Net or some Python applications or PHP applications and the people developing them need access to the IBM I data. And rather than taking each use case one at a time and giving them access to what they need, they’re just granted generic access to the database via odbc. And so now I have credentials off my IBM I that have very, very high levels of access to the IBM I data. We’ve also seen some ransomware attacks that have affected mapped IFS drives. So they get to your IBM I through a map drive on the ifs. And then the other danger are injection attacks where people are attacking through your web, through your web pages. We see very, very few IBM I customers who have integrated their IBM I systems into their multi factor authentication enterprise strategy. So now we don’t have that same level of security for the IBM I. Then a really common thing we see are inadequate controls around DevOps where the IBM I customers actually don’t know what source code created the objects that they’re running in production. So they have objects running in production and they say all these source code is in library A. But when you actually look at the object says no, I was created from Dan’s development library or I was created from some temporary library. And so now I have something in production where the source code that I think it was created from isn’t matched up with the object. So it may have been created from that, but it may have been created from something else. And then there are, there, there, there are sometimes holes in the process of how do things get into production, who has to actually look at it, who has to actually approve it. So these are all things that can, can create vulnerabilities in your system. So how do, how do the credentials get compromised? So how does that happen? So almost all attacks start with compromised credentials. That is the most common way that systems are being attacked. And how does that happen? Well, it happens a few different ways. One is this. These are phishing attacks. That’s where you get, somebody gets an email that looks very official. So this was actually one that came out a few years ago during the pandemic. It looks very official, everything looks fine. And there’s Just this little link down here at the bottom that seemed harmless enough. If you click on it though, it downloaded a key logger on your system. And the key logger allowed the hackers to actually record as you put in your credentials as you signed into something so they could get a hold of your credentials that way. And I just got a really, really official looking one. I was actually waiting for a contract to come in from a customer from docus and I got this email coincidentally at the same time. It looks very much like a DocuSign email, but the links, all the links are wrong. It’s got bad links throughout. So it is a phishing email. But it was, I was very, very close to actually clicking on it because I thought, oh yeah, it looks like a DocuSign email. So phishing attacks, very dangerous. They’re get, they’re, they use a lot of social engineering to try to, to try to trick you into thinking it’s from somebody you know and to get you to click on it so that they can download the software and get your credentials. And then the other way that people get credentials is because of the ubiquitous use of basic authentication. Basic authentication means that I am, what I am using to authenticate to my IBM I is the IBM I user IDs and passwords. So I’m giving people who are coming into my system native credentials to my IBM I. And if you’re using basic authentication, one of the problems with basic authentication is that it’s not encrypting by default. It doesn’t actually encrypt that information. It encodes it in base 64 encoding. Base 64 encoding isn’t encryption, it’s just encoding and it can be easily decoded. So if somebody can intercept that communication, they can get your credentials. And so I’m giving these native credentials to the outside users. Now we have some customers who use VPN access to try to get a little bit more security around that. But that becomes a problem. If you really want to give your customers and your partners access to your system. You don’t necessarily want them to have to have VPN access to your IBM. So we see this, it’s very, very common to be using basic authentication on the IBMi. Right. Is if you’re using IBM I user ID and credentials, you get all of the security that you’ve set up automatically. So if you have that profile restricted in certain ways that there are certain things it can do. Well, you get that because they’re signing on with that credential. But they are, whoever is Using it is getting native IBM I access.

Aaron Magid 10:06

Right. And Dan, if I can just jump in for a second on that to add to it, because you mentioned VPNs and we’re talking about these security policies, it’s really important to understand that just because you’re inside of your company network does not mean that the communication is secure. Right. There are the. Right. I don’t know. There are studies that show that the vast majority of hacks originate on internal company networks. Right. Meaning what happens is you get an infected device on the company network. And because a lot of companies have this sort of castle and moat approach to their network security where you just have sort of this one layer where outside of the network nobody can get in, but then inside the network everything’s unencrypted. And using basic authentication and using user profiles for authentication, what happens is if I can get something onto your network that has an infected, that, that is infected, at that point, I then basically have free reign over the entire network. And so it’s, it’s a very important point here that you, you can’t assume that the network is secure or that the other machines are trustworthy. So I just wanted to make sure to mention that because that’s often. I think one of the justifications that I hear for basic auth is people say, well, you know, I’m talking from my server to another one of my servers. I know I trust it. But the truth is you actually can’t trust that. And that’s, that’s a mistake that a lot of hackers exploit.

Dan Magid 11:35

Yeah, exactly. I think gartner said that 67% of attacks originated inside the network.

Aaron Magid 11:41

That’s the number I was trying to remember. Yeah, I remember it was in the 60s.

Dan Magid 11:44

Yeah. Anyway. And people are using just HTP instead of HTTPs. Well, now that means that I’m just, I’m just sending stuff in, in plain text. And again, that’s usually because people say, well, it’s on my network, I’m safe. So you really want to get to HTTPs where you, where you’re not vulnerable, where somebody gets in and listens in the conversation, they still can’t read the data that’s being sent up and down the line. So some of the reasons why you want to start getting away from basic authentication, and I already talked about that. The credentials are 64 bit encoded, not encrypted. By default, the credentials are being sent on every API call. So every time you make a call, there’s a credential being sent. And that means that many, many Opportunities for somebody to discover it, discover those you can only use username and password. You can’t use biometrics, you can’t use other methods of doing, of doing security. The the native credentials are stored wherever the client is. So if the client is, is outside of your network and a system you don’t control, you now have your credentials on somebody else’s machine that, that, that, that also has the information about where your machine is. And so they, a hacker who hacks their machine can get in potentially into your machine. They don’t have a built in expiration method. So there isn’t a method built into basic auth to say these credentials are only good for a particular time and they don’t have built in granular permission controls as to what specifically you can get. And again you can take advantage of what you set up for that user profile on the IBM I. But the basic authentication itself doesn’t provide for that. And almost everybody is for those reasons deprecating or getting rid of basic authentication. You look at all the major vendors, Microsoft, Google, Apple, the big API vendors like mulesoft and Apogee, they’re all saying we will no longer support basic authentication because of those of those issues, because of the insecurity of basic authentication. So let’s talk about some of the nightmare scenarios for the IBM I world. If you have the situation where you’ve got native credentials that are off your system and those credentials have some kind of high authority or even something stored on a programmer has their own credentials and they’ve got high authority and weak password rules. Those provide opportunities for somebody to, to get those credentials. And once they have them of course they get all of those authorities so they have unfettered access to your database. We see this a lot with customers who have NET applications or something. They’re talking to the IBM I and they just give the non IBM I systems ODBC access to the database and they can do whatever they want. Well if that authentication, those user IDs and passwords get compromised, now somebody outside has that unfettered access to the database. The other way people can get in is they can use a SQL injection attack to hijack a connection to the IBM I. If you’re giving somebody an API access or you’re giving somebody web access to your application, any hacker successfully injects a SQL attack into that communication, they then can hijack the connection and use those credentials to get at your IBMI nar. I didn’t know if you wanted to add something around that.

Aaron Magid 15:22

Right. So well, this is a really important thing, I think, you know, when we talk about these, these kinds of scenarios. This is one of the most common scenarios that I see because I do a lot of integration work with IBM I shops and probably the most common vulnerability that I see out there is this sort of high, massively over provisioned profile being used for access remotely to the IBM mind. And you know, this story may resonate with, with a lot of you here. I usually, I usually most shops that I talk to have, have had this experience. But basically what I’ve seen working with lots and lots of shops is a standard story where one day you’ve got a Net team or a Java team or Python team or a Node team that wants to get into the ibmi. They get an ODBC access or jdbc, they get their connection, they set up their connectivity and they start running queries. And the IBMI team gives them a profile that they can come in as their application is storing the credentials to this profile on a Windows or a Linux server that is usually going to be in your network DMZ that’s sort of, you know, a little farther out there. It’s not as protected as the ibmi, right? Because it needs to be servicing their web apps and it’s, it’s got those credentials on there and it’s talking to the ibmi, okay? So the developers are working on it and eventually they run into permission errors because the original profile they were given doesn’t have access to some production table, okay? So they run their query, they talk to the IBM I team, they get the access they need, then they run another one, they run into another error, they talk to the IBM I team, they get the access they need. Eventually, after going through that loop a bunch of times, what I usually see is the IBMi team gets annoyed from having to answer all of these permission requests all the time, or the external team gets annoyed and so they just end up granting the profile all object authority. I see this on almost every IBMI that I get into that. You’ll see the profile that those QZDSO INIT jobs are running under on the ibmi and that profile has got all object authority, right? So the external application has that authority because they didn’t want to deal with the permission requests. The problem with that is that now what you have is a server that is farther out from where the IBMI is that is less protected than the IBMI that has credentials to the IBMI that go to an all object authority profile, right? So if those credentials get Exposed, we have a huge problem. And also critically, this is where it gets even more effective from the hacking perspective, is if that application has a SQL injection vulnerability, which I see all the time as well, meaning if that database connection can be hijacked and used to run other commands in that. In that SQL query, which is easier than you might think and much more common than you might think, that means that your web app is exposing a hijackable connection to an All Object authority profile on the IBMi. Right? And when you couple that with the IBMi’s capabilities in the database, where DB2 has stored procedures for basically everything on the system, that means that that database connection with All Object authority has the ability to do things like retrieve any data from the database, like ship out data to external systems, right? To grab your data and send it to someone else using the SQL HTTP functions that are available to that database connection, or go out and grab ransomware and pull it down onto the machine through again through those same functions that are available to the database connection. So again, it’s a situation that is. Again, I think Daniel hit the nail on the head when he mentioned these systems largely rely on security by obscurity. Right? Meaning hackers often don’t know what all Object authority means and they don’t know about this relationship between the database systems. And so they don’t know necessarily that they can jump into your. Net application, grab the credentials, and now they have an all access pass to your IBM I to do whatever they want. Right. But I see this in almost every shop that I talk to. It is almost everywhere. So it’s a critical thing just to keep in mind as you’re looking at these, you’re to look at the access to your IBM I and make sure that even if your security protocols are effective, that the people who have access to your system are also using it in an effective way and that they have multiple layers of security there.

Dan Magid 20:16

Great, thanks, Aaron. Just a couple other things and then we’ll actually talk about solutions. This is the thing I mentioned earlier is that weak DevOps control creates vulnerabilities. That is, you could be running objects in production that you really don’t actually know what’s in them. Where you’ve got an object that says I was created from Dan’s development library, or I was created from a temporary library, but in the source library where the the source is supposed to be, I don’t have that source code, or I have a different version of that source code, so I don’t actually know what source is Created because maybe Dan’s library is gone or the temporary library is gone. Those things don’t exist anymore. So I don’t know what’s in that object. And then do I. As changes get made and they move through our life cycle, is there a process where somebody takes a look at that code? What, what code was just put into the system? What code are we moving into test? What code are we moving into qa? What code is going into production? Do we scan the code for vulnerabilities? Are we looking for vulnerabilities in our, our DevOps process? Do we run automated tests of the system? So are we actually looking at that code that’s getting through our life cycle or are we basically saying, you know, all our developers have been here for 30 years, they’re all trusted and I’m not going to worry about those kinds of things again. The problem is if somebody gets a hold of compromised credentials on their system, they can get in and they can make source code changes. If I have somebody who compromises the credentials on my machine, they can get in, make some source code changes, or add a module to something, stick it in my test environment. And if nobody’s ever looking at that code, that code can get promoted and created and deployed without me even knowing about it. I don’t even know that I’ve got that thing going on in my code. Again, that’s the thing that happened at at Target where they had code that was promoted through their process and deployed to all of their systems automatically because nobody saw the code that was, was added into their system. So this is absolutely something that can happen. And if you don’t have those sort of tight controls, you’re vulnerable to these kinds of attacks. So let’s talk a little bit about. So those are some of the issues. What are some of the things that you can do to, to protect yourself? Well, one of the things is, I like to say assume your credentials have been compromised. What would you do if that had happened? I actually went to a security presentation where the presenter was guy who had been doing security for like 30 years, said just assume your system has been hacked. He said, don’t worry about it anymore, just assume it’s happened and make sure you’re taking steps to say, if somebody has my credentials, how am I protecting myself?

Aaron Magid 23:02

So actually Dan, if I can just jump in for one second here. That, that actually, that message, I got the same message when I was taking my, my security courses in college. They, they told, they did a very similar thing. The unit on prevention in These security classes was like a couple of weeks maybe, and then the rest of the course was all about detection and remediation. Right? Because the assumption is that there are a million ways to get in and we have to make sure that we know what to do and how to block, you know, and how to effectively recover and how to make sure we’re monitoring, how to make sure we have the processes that we need deal with it.

Aaron Magid 23:44

Right.

Dan Magid 23:46

So yeah, so basically, yeah, so, so, so stop using basic authentic authentication. Start using things like token based security. Implement a zero trust policy, which means you’re only providing people with just the access they need. Not here’s, you know, ODBC access to the database, but you get access to these rows, these columns in this particular table for what you’re trying to do. It requires more effort, but it protects you. And the other is to create multiple layers of security. One of the things you want to be able to do is say, well, if they get through layer one, they’re going to run into layer two. If they get into layer two, they’re going to run into layer three. So you actually layer your security to make it so that a hacker would have to get through multiple levels. The idea is you want to make your system harder to get into than anybody else’s system because what people will do is they’ll go to the, they’re going to go to the easiest systems to break into. So if you can make yours harder than everybody else, they will avoid trying to get into your system. Add multi factor authentication to your, to your security strategy. Third party authentication, you know, using something like OAuth 2 to get third party verification. That person is who they say they are. And then in your DevOps process, add code reviews and approvals and other ways of looking at the code and then monitor what’s going on, keep track of what’s going on. So let’s take a look at some of those things. So again, old way send in native credentials. The new way use token based security. So tokens allow you to create an encrypted token rather than native credentials that identifies the user. And even if somebody gets a hold of the token, they can’t read it. It’s an encrypted token and the token expires. So there’s a limited amount of time that it’s going to be able to work. And then there can be all kinds of rules of what somebody can do with that token. So if you look at a JSON web token, one of the kinds of tokens the token identifies who you are, it identifies what you can do. It actually says, here’s what this person has the ability to do. And then there’s a hash that proves that it’s a valid token, that this is a valid message. This is what a JSON web token would look like. It’s a JSON web token. This is the hashing algorithm it’s using. Here’s the actual load, this is the actual data that it’s sending, and then here’s the hash signature to make sure that this is actually a valid message. That’s basically what a JSON web token is. Now, one of the things that you can do, one of the things we’ve done at Erdani, is make it possible to easily take advantage of this technology. So I’m just going to give you a quick example of that. So here, this is the Erdani workbench. And I can go in here. I’m going to go ahead and generate an API right here that’s going to be a query against my customer database. It’s going to just get all the customers from my customer database. So I’ve got this API and I can generate the code for that API. Now we’re going to generate all the code for creating the API endpoint. We’re going to generate all the code to do the data transformations from things like JSON into RPG data structures or into parameter data. We’re going to generate all the routing code. But this is going to create basically a basic API that will do all the things that you might need it to do as far as getting the data from that customer file, it’s now created that data. So it’s created that API. I could go out to the web and hit this API and get that data. But what I want to do is I’m going to say, you know what I’d like you to add to this code processing or JSON web tokens,

Dan Magid 27:38

so I can tell it to add that. And now what it’s going to do is it’s actually going to add to my API the JSON web token processing code so that I don’t have to write that code, I don’t have to learn how to do it. I can just go ahead and generate code for adding JSON web tokens to my API. You can see here. Now it’s adding, it’s actually adding that code here to my, to my API code. And it actually does this in a couple of different passes because it not only generates the code, then checks the code for errors, and in A second pass, it then runs it again and it runs another scan of the code. So it’s running a variety of scans of code to actually create the end code that you need in order to process this. And then you can get in, since this is just standard JavaScript, if you want to, you can get in and you can make modifications to this code, but it gives you a way to create that support in your APIs.

Aaron Magid 28:39

Actually, Dan, if I can jump in for one second in there, I think one of the other important points here is the JSON web token support that you’ve got set up there is creating JSON web tokens and using them. And that’s great. It’s also really important to note that you’re also not using IBMI credentials for authenticating against your API, right? Meaning if you’re using IBMi credentials, meaning an IBMI user profile and password as the, as the credentials to the, to your API layers, going up to JSON web tokens from basic AUTH is better because then you’re not storing credentials all over the place. You only need to enter them less frequently, depending on how long your expiration interval is on those JSON web tokens. But it doesn’t get you all the way if you’re still using IBM I credentials. Because even then with JSON web tokens, you still have to send in your IBM I credentials, you still need to keep track of those. And all of your users would still need IBM I credentials somewhere that they can send in to get their token back. And that’s an opportunity for those credentials to be exposed. And those credentials because they are system credentials, right? That’s the key point. Because they are system credentials, if they are exposed, they offer much more access than the API might allow, right? Meaning within the API I’m allowed to do certain things. I can grab customer data, I can update records, whatever, that kind of thing, whatever the API lets me do. But if I actually just have the credentials and I can get a network connection to the IBM I, I can open a 5250 session, I can open an SSH session and I, I could connect directly to the database and suddenly I have access to do anything that that user profile is allowed to do, which again, because of what we’ve talked about, is often largely over provisioned and is able to do lots of things that it’s not supposed to be able to do. So another key point here is to make sure that we are not using IBM I credentials to do the authentication here. That is something that we wouldn’t do on any other system. Honestly, the IBM I think is the only place that I’ve actually seen APIs that do that, that have system credentials used for API access. I don’t think anybody else does that. I think it makes sense why people did it originally on the ibmi. But it’s something that we really do not want to do if we’re trying to keep our, if we’re trying to keep our system secure. It’s a really, really not the thing we want to do. Right.

Dan Magid 31:13

All right. And so we talked about multiple layers. So I get the token based authentication. Get away from sending out IBM I credentials. The next thing I might want to do is add multi factor authentication to my environment. So multi factor authentication for anybody who’s not familiar with it, I mean everybody, I’m sure everybody uses it. It’s a combination for authentication of something the user knows, like a password or a secret, something the user has, like their phone or a fob or an authenticator app, and something that is the user. So that would be biometrics, like a fingerprint, like a retinal scan, like a voice print, something like that. And having multiple factors in order to be able to authenticate. And what you can do is you can, there’s, there are lots and lots of providers and probably in your organization you already have multi factor authentication running for some applications. And so one of the things you want to be able to do is tie your IBM into whatever your enterprise multi factor authentication strategy is. So a lot of companies are using Cisco’s Duo for doing multi factor authentication or Microsoft Entre or Okta. You can tie into those systems and say, okay, I’m going to do multi factor authentication the same way. I just have to be able to do it with my IBMi. And maybe I want to do that with a 5250 session. So that when you sign into 5250, you don’t just get right on the system. It sends a message to your phone or something like that. You have to respond. And so I think Aaron, you have a, you have a demonstration of that, right?

Aaron Magid 32:47

I do, yeah. So this is, this is one of the things that we always want to show because it’s something that there are, there are a lot of, there are a lot of ways that claim to be able to do this on the 5250, but there are usually some pretty notable restrictions like for example, having to use a specific MFA provider or having to have a separate MFA system for your IBMi as opposed to everything else in your organization. So one of the key points here, and this applies to everything we talk about from a security perspective, is we want to make sure that everything we’re doing is fitting with industry standards. And the main reason for that is that security is an arms race, right? We are constantly fighting, constantly racing against hackers. They find a problem, they hack into the, you know, whatever the, the U.S. treasury last December, right? They break in and then we patch the vulnerability and then they find a vulnerability and then they hack in and then we patch the vulnerability. It happens over and over and over and over again. If you are not using industry standard tools, you are not benefiting from that arms race. Which means what happens is hackers find a vulnerability, they break into a system, that system gets patched, but you don’t get the patch. And so then over time, your system falls farther and farther and farther behind on security until eventually a hacker realizes that you’re there. They come in and they say, oh, wow, yeah, I’m going to exploit this vulnerability from 1995 that your system still has and I can get into your system. Yeah, it was patched for Everybody Else in 1995, but it doesn’t matter because you didn’t get the patch. Right? So there’s a critical piece here. We want to make sure that we’re using the same tools that everybody else is using so that we can be lifted by that same tide that is bringing everybody else up. That same, that same war, I guess I should say that’s, that’s happening. So I’m going to share my screen here and I’m going to show you this. So I have Duo on my, on my phone here. I’m going to use Duo as my MFA provider. That is an industry standard MFA provider, right? So this, this system here, I’ve got it right here on my phone. My application that you can see right here is doing my multifactor, my multi factor codes that I can use to get into my IBM. I. So what I’m going to do here, this is a key point. If we want to mitigate some of these issues around user profile security, right? If I have user profiles that have access to my system and if I absolutely have to give user profile access to my people or to my, to my other users within my company or to end users, we want to make sure that that access is secure and that it’s not being used by the wrong person. So one great way to do that is multi factor authentication. So I’m going to do here is I’M going to come in, I’m going to say mfa, I got my MFA user but put in my password, right? So I go into login and what’s going to happen here is my phone is going to pop up and it’s going to say, hey, is this you? Are you sure that you want to approve this session? I’m going to say yeah, go ahead. And at that point my 5250 is letting me in. Right. So in this process we know that it’s me. So you know, obviously it’s still not a great thing if my credentials get leaked, but if those credentials are exposed, we actually have a way to tell, we actually have a way to block access to the system if it’s coming from the wrong source. And just to prove out that I can actually do that, sign off here, do this again and over here I’ve got my push notification over here I’m going to go like this instead. And no, I don’t like that one. And I’m going to say not suspicious so that it doesn’t send off, set off any alarms. And at that point I get kicked back to my sign in screen. Right. So in here again I have multi factor authentication on my sessions so that since I, I have to be using these, if I have to be using these credentials, I at least can make them a bit more secure. I can at least help out with this, with this process and keep my systems a little bit, a little bit more locked down.

Dan Magid 37:05

Great, thanks Aaron. Let me go back and share. So again, that’s another layer. So you know we talked about using, using token based authentication. We talked about, we, we talked about multi factor authentication. Another layer you can add is, is third party verification. That is, I’m going to have a third party verify that this person is who they say they are and they can do the authentication for me. So this would be using technology like OAU and talking to providers like OKTA or Microsoft Entre who are going to verify that user for you. So this is using this technology you see often this is, you know, the sign into Trello, you can say you can, you can sign in using Google or Microsoft or Apple that’s using an OAuth 2 kind of a connection where what you’re going to do is authenticate to that particular provider and they’re going to provide you with an identity token and then you’re going to validate when that user comes back to you to get something from your IBMi, you validate that, that’s actually a valid token that they’re bringing with them. So this is kind of what that flow looks like. I know it looks kind of complex, but basically the user initiates their connection with your client act. The client app then says, oh, you need to go authenticate. I’m going to send you over to Okta, for example, in this example. And then Okta says, okay, give me your credentials. If the credentials are, are not valid, they send you back to the beginning. If they are valid, they say okay, you’re authenticated. Or they say wait, am I requiring mfa? Because Okta can say, no, you need to do mfa. And if it is required, they then prompt for the Multi Factor Authentication. If that’s successful, then they issue you a, an access token. They issue something that identifies you and says this is a valid user. You then get redirected back to the resource you were trying to access. That resource says in this case the IBM I do you have a valid token. If you do, then I will give you access. These tokens like the JSON web tokens we talked about earlier, identify what you’re allowed to do. This gives you what are the authorizations. And these tokens again can expire so you can make sure that they don’t last long. You can have all kinds of rules for, for how somebody gets these tokens. So basically, whoops. So basically you can add this. Now again, this is code that, that we can, we can help generate for you. So you don’t have to write all the code to do this. You don’t have to try to write an RPG OAuth processor that, but we can generate this kind of code for you. And then the other thing you can do is you can add the IBM I to your single sign on environment. So most of your companies probably have something based on Active Directory or Entre or they have Kerberos or saml. They have some system for doing single sign on. Single sign on means I sign on in one place, I get an authentication token again that says this is who I am. Something I carry around with me that says this is who I am and I can then present that token instead of more credentials in order to get into other systems. Now a lot of times the IBMI is not included in this single sign on kind of an environment, but it can be so you can set it up so that you don’t have to, once you sign into the network, you don’t have to re authenticate to the ibmi the system already knows who you are. And this is kind of what that looks like in a Kerberos being one of the single sign on technologies. This is kind of how that works. The you sign into your PC, it contacts the Kerberos authentication server, the key distribution center. If Kerberos authenticates you, it sends you what’s called a ticket granting ticket, which basically just says, I know who you are, I’ve authorized you. And that has an expiration date, so you can only use it for a particular amount of time. And then you then request the specific thing you want to be able to do. The system checks to say, are you allowed to do that? If you are, it then gives you another token to say, okay, you’re allowed to do that particular thing. You get a service ticket to say you can perform that particular function. The system then validates that service ticket. And if it’s valid, you get your access to the specific thing you were granted access to on the IBM I. Again, so you’re not getting like generalized ODBC access, you’re getting access to specific rows and columns in a table, or specific program functions, or specific, specific queries. But you don’t have to re authenticate. Once you’ve done this process of getting your ticket granting ticket, the system knows who you are. And when you request services, it goes to check to see are you allowed to use that particular service. And again, in our upcoming sessions, we’re going to go into these technologies, into some more detail and how you actually implement them, if you want to add them to your, your IBM I environment. And then finally the thing you want to make sure you’re doing is if you are giving people access to the system, you want to be able to monitor those connections. So you want to keep track of who’s getting into your system. What are they doing? Are there errors, are there suspicious attacks? You know, did somebody’s the number of connections or the number of calls that somebody’s making your system go from, you know, 50 an hour to 10,000 an hour all of a sudden, did you see the sudden spike in traffic? If so, you know, maybe you want to get notified, maybe you want to get some kind of an alert to say that, hey, I want to know when something like that has happened. Am I under an attack? Is there, is somebody doing a distributed denial of service attack? Am I suddenly getting lots and lots of communication way beyond what I would normally get? So you can monitor those connections. And one of the things that you want is you want to make sure that all the, all of the things that you’re connecting with, all the logging is being done in standard prometheus format. Prometheus is a standard logging format that any of the dashboarding tools can use. So you can use Splunk or New Relic or Datadog, Loggly, Grafana, Swagger, all these different kinds of dashboards can read that data. And then you can create your own specialized widgets for the dashboard that you want to use for keeping track of who’s trying to get into your system. And you can do things like throttling. So you can say, you know, if suddenly traffic spikes from a particular site or overall, I want you to cut back on the traffic, I want you to throttle those requests as they’re coming in, or I want you to block these addresses or only allow these particular addresses so you can, you can control all the communication coming into the IBM I and monitor it and get notified when they’re, when there are problems. So Aaron, I think you’ve got a monitoring demonstration. I do.

Aaron Magid 44:03

I think it’s always best if we can, if we can actually show the things that we’re talking about instead of, you know, just talk about them. Though I, I think your explanations are wonderful. I’ll share my screen. So this dashboard that I have here, this is Grafana. So this is one of the dashboarding systems that, that Dan was mentioning and it’s using a lot of those tools that he was talking about. So again, industry standard tool available for all platforms. One key point here, this application, this, this monitoring application can pull in metrics from everywhere. It can pull in metrics from our RPG programs, can pull in metrics from our database, it can pull in metrics from our SQL servers and from our open source systems, from our node applications, Python.net, java, whatever we have going on. So again, one of the key points here, in order to be able to secure your systems, you have to be able to see what’s going on in your systems, right? And so that observability is a key point to be able to, to see what’s happening, address issues as quickly as possible before they become serious, serious breaches. So one example that I wanted to bring up here is this dashboard is related to the integration that I was showing before to my MFA system. So it says, okay, you’ve done five MFA calls in here. That sounds about right for what I’ve been doing with this, with this demonstration and the testing that I was doing, setting up before, before we did this webinar. But I want to point out here we can pull high level numeric data and we can look for trends and patterns and things that are happening in our application. But I can also get some detailed data about this system. So for example, down here, if we look a little bit lower in here, we can see what’s going on in my MFA system. I know this is a little small on here, but if you notice, I got these two that are colored in green and these are two allowed MFA calls. And I can see who was authenticating, it’s me, whether or not they allowed it. And I can see when they were authorizing into my system, right? So that’s all available in here. And then I can also see over here, marked in yellow, here’s one where the MFA was denied, right? So this is something that I would want to be watching for, right? This is a, this is a scenario that either I’d have a person watching, but I’d also probably set up alert conditions in my dashboarding system to watch for this. Because this is saying, hey, there was a sign on session to my 5250 where somebody pulled up their phone and they chose to deny the session. Now, okay, we all make mistakes. We might have hit the wrong button to hit the red one instead of the green one. But that’s an important thing to be watching for in our, in our applications and to be alerting on it. Because the data is in my dashboard, I can see it and I can also set up alerts for that data, right? So again, this is a critical piece. And another key point here that, that I love in these dashboarding systems is if I want to look at this particular log entry, I can go over here and I can say, show me the context. And it’s going to show me in my logs what was happening around the time that that particular warning came up, right? So this is a very powerful feature of these kinds of tools. When I am looking at an error, right? Oftentimes when hackers are trying to get into your system, not if, when you’ll see lots of errors, lots of warnings, lots of validation errors happening in your system. One of my favorite things that I see when I work with companies on their APIs is they’ll get, I get this call a lot from companies I work with. They say, I see all these weird API calls coming in. I just got this flood of API not found errors in my system, all these weird URLs and they send me a screenshot of it and thousands of these calls that are like, get, slash, login, slash, password, secret, slash, give me all the passwords, right? They like, there are all these URLs that are not things that they’re providing and the system is rejecting them over and over and over again. Well, what’s happening there? It’s actually very common, very commonly a company security scanner nowadays. But it also is very commonly hacker bots. So you’ll have a bot out there that sees that you put an API up. They don’t know what you’re using, they don’t know what’s available, but they know there’s lots of known vulnerabilities out there. So what they do is they start hitting your API with all kinds of calls that will exploit known vulnerabilities. And what the bot does is it says, if I get a hit on this, I’m going to notify my master, right? The actual hacker, that, hey, I found a server that’s still running a version of an application that has the log4j vulnerability, for example. So they’ll try that one out, they’ll see if they can get a result from it. And if they get a positive, then they say, hey, I found a log for J. Come break into this system, right? So we see that all the time. So it’s actually a good thing when they see that in their API logs, that they’ve got all those, those failures, meaning they get all those messages and then they’re. And they’re all getting rejected. It’s an important point here that oftentimes when someone is looking to break into your system, there are going to be patterns in the logs, there are going to be errors. For example, they try to call your API, they try to do some SQL injection, Your API is probably going to reject the first call, right? Because they’re going to do it wrong. It’s going to fail on the database query. So you’re going to get a database error. It looks like you just had a failed API call, but if you actually inspect it, you can see that somebody’s trying to break in. So being able to see this context is very important because we can see not only is there an error, but I can scroll up in the context and say, hey, wait a minute, that input looks really weird, right? If I see like quotes and semicolons, dashes, right? I can see that there’s a problem there in the data that they’re sending in. So again, just critical pieces of being able to observe our systems. Now, I wanted to show you an example because I actually have some alerts set up in this application. So what I’m doing here is I’m generating a huge. I, I just hit a. Started an automated process that is basically doing A denial of service attack on my API. So it’s just hammering it with requests over and over and over again to my EDI endpoints. For anyone who does edi, what this is going to do is run hundreds of documents per second through this system, which is an outrageous amount to try to do in most integrations. And so what I’m going to do here is I can actually bring up my, my slack here. And I actually have warnings coming from my application. You can see it says here, firing EDI, too many requests, EDI alerts, it says 404.916 calls per second, right? So I’m actually getting an alert directly to what my operational center is, which for me is slack. So I’m actually getting an alert directly to my application saying, hey, something’s happening here. And I can see, oh, actually this is the, the one from now that’s actually from earlier. So this one is the actual alert from now. Sorry, it’s actually 362 calls per second. Right, but that is my alert coming from my dashboard. You can actually see that in here. If I go to, if I actually got one second here, just change that. If I actually go here to my EDI dashboard, I’m getting all kinds of messages, 45,000 112,000 documents coming through here. But I’ve got all kinds of things happening. And I can go to my alert rules and actually says one of these is firing. I’ve got an, I’ve got an alert rule that is firing, saying, hey, something’s wrong here. And if I come back here and I say, okay, let’s shut down that process, it’ll take it a couple of seconds to revert back. But what I’ll get is a message along these lines that says resolved, that says, hey, okay, now our average has dropped back down below those numbers for the alert threshold. And so I can, I can rest, right? And this application, just to, just to point out, if I wanted to build a, an integration here, if I wanted to add a contact point to my application, to my dashboard, this thing supports Grafana, supports basically everything, right? So I can send, I can send things to Discord, I can send an email, I can send things to, to an API endpoint, I can send a teams message, I can go to pager duty. And all of those things are available in, in this application, right? To pull from my dashboards into my applications and show me exactly what I’m looking for in my, in my application. So I can actually see what’s going on and be alerted based on automated conditions in these processes.

Maia Samboy 52:47

Just wanted to cut in for just a second and let you guys know we got about five minutes left. So I didn’t know if you want to do pause for.

Dan Magid 52:54

Yeah, I’m gonna, I’m gonna wrap up real quickly and then we’ll, we’ll go to. I know we’ve got a couple of questions out there, so let me, let me just wrap up just the last couple things here and then we will, we’ll go to the questions. So let me just share real quick.

Dan Magid 53:21

Okay. Yeah. So this is a little bit about how Erdani can help. So the purple box here is Eridani Connect. And it is a system that allows you to generate APIs with the kind of security things that we’ve talked about with JSON web tokens and with third party authorization. So we can help you generate that to make that process easier, to do the blocking, the throttling and to do the monitoring and then passing your connections through on a variety of connectors to the ibmi. And it can do that both on an inbound basis and an outbound basis. So we can provide that kind of security for you either way. And then we can also help you strengthen your DevOps environment. So if you want to make sure that everything that goes to production has been reviewed before it gets there, that you’ve run your scans on it, that you know where the source code is for all of your objects, we have things that can help you do that. So there are several things that Erdani can do to, to help you with, with your process. And you just need to, you know, just reach out to us and we’re happy to talk to you about that.

Aaron Magid 54:22

Naron.

Dan Magid 54:23

I know, I saw, we had a couple of questions. One, one from our good friend Mike Corbo. You saw Mike’s question?

Aaron Magid 54:32

Yeah, yeah. Well, I think I, I answered Mike directly in there. But, but yeah, Mike has some, some, some requirements in there that he’s been bringing up for, for, for doing MFA on the ibmi. And, and I think, Mike, what you were bringing up is a very important point, which is that you, you have to strike this balance. It’s a very difficult balance to, to hit, right, because sometimes your MFA provider is down. Hopefully that doesn’t happen very much. But I actually had a, I was actually doing this demo for somebody with, with Duo a couple of weeks ago, and Duo actually had a brief outage course while I was trying to do the demo, which was very fun. But that, that can happen. And so we need to make sure that we have multiple paths, but you need to make sure that when you’re creating multiple paths in and when you’re creating your own back doors to make sure that you can get around outages in duo, that you’re not creating a back door for somebody else to use. Right. So there’s a critical process there that we need to make sure that any piece of our system is, is effectively secured, though we do need to have backups for, for each mechanism.

Dan Magid 55:40

Yeah. Then I notice you were, I think you’re typing an answer to Jonathan about the question about.

Aaron Magid 55:46

Yeah, so yeah, why don’t I answer this one? Why don’t I answer this one live?

Dan Magid 55:51

So what are your favorite methods for preventing injection?

Aaron Magid 55:55

Right. So in terms of preventing injection, I would say. So I actually do like using parameterized queries. I think that that’s the best thing as opposed to ORMs. The concern that I have with ORMS is what they. For anyone who’s not familiar, an ORM is an object relational mapper. Basically what it does is it allows you to say, get me the customers and then it understands your database so it will generate the appropriate SQL for that actual query. Well, think about orms. Number one is they’re generating SQL for you and you don’t actually necessarily know how they’re generating that SQL. So that for me is always a bit of a concern. SQL is code, right. So when I have a system that’s generating code that then gets executed at runtime, that I cannot see that gets concerning to me because if that ORM does something wrong, it’s sending instructions to my system and my system has to blindly accept it. So that’s, that’s why I, I generally don’t use ORMs. I generally will do parameterized queries. And I would say if you’re using an orm, it should be using parameterized queries. I, I generally don’t feel the need to jump over to stored procedures. I think queries are very strong as long as you’re using parameter bindings. And actually just to give an example of this, I, I want to make sure that, that I show the concept of SQL injection here. I know we’re very long time, but just so I’m just so that everybody is clear here, to give you an example, what SQL injection does is it basically hijacks a valid query. So to give you guys an example here, let’s say that I have a query here that says our base query is select STAR from QIWS qcuscdt. You can test this out because every IBM I has this file. And then what we do is we say, okay, we got to take in a customer ID. So what we do is we say plus customer ID, we stick that into our SQL query. Well, if our user puts in 1, 2, 3, 4, 5 for their customer ID, what we get is this SQL, which is great, that’s valid. Select the record where customer number is 1, 2, 3, 4, 5. Problem is, what if our customer puts in this value? This is one quote, space or space one equals one, and then a double dash. What that does is it generates this query which says, give me the record where the customer number equals one or where one equals one. And then the double dash is a comment, so it ignores the rest of the query. And I actually have an API, I can show you this. Actually have an API here that I set up to show this. So if I pass in a valid customer ID right in my API here, actually only one second and I got my oops, one second here.

Aaron Magid 58:48

You know, I may or may not have the, the time to set this up here, but if I go in here and let me just, let me actually send my query, you can see that if I send it a customer id, I get that record back. I’ve got just this guy, just this, 938472. But if I take that value that I passed in here, let me just grab this out of my little notes document here, and I paste this in here. So my API call looks like this, and I send that call. Then my API returns all of the customers in my database, right? So that’s what SQL Injection does, right? And this is a very simple case again, remember, because the IBMI allows you to do things like call APIs from stored procedures, like interact with the file system from stored procedures, like run any command from stored procedures. If I can hijack that SQL query with a more complex input here, I could do things like modify some source code on your system, compile it into your production system. And because of what Dan talked about earlier, where most companies actually have no idea what source is running in production, if I do that to the right object, if I do that to an object that’s old enough, you will have absolutely no idea that it happened, right? There’s no trail, there’s no trace. I can get in and do whatever I want because this profile has all object authority. I can run whatever query I want, I can modify the programs, and there is no audit trail. It is a hacker’s dream, right? So all those things together, we talked about all object authority. We talked about having auditability on your source code and making sure that we know what’s running and enforcement on it. SQL injection, those things come together and you have what is truly a nightmare. And again, all of those things are present at almost every IBM I shop that I have ever talked to. The SQL injection is less common, but the other pieces are there. So anyway, just a key point here. So the key way you get around this is that parameterized queries that Jonathan was mentioning.

Dan Magid 1:00:51

Okay, great. I mean, Kev, to cut you off. I’m sorry, a couple other questions, but we will respond to those in writing. And Maya, I will leave it with you to just talk about the next session.

Maia Samboy 1:01:03

Yeah, thank you for coming everybody. We will follow up with the questions we didn’t get to today. I put it in the chat. But I just wanted to let you all know the second webinar in this series, Beyond Passwords, Modern Authentication and Authorization, will be Tuesday, April 8th at the same time, 1pm Central. Keep an eye out for that email invitation as well as the recording from the session today and I hope to see you all in a couple of weeks.

Aaron Magid 1:01:29

Great.

Dan Magid 1:01:30

Thanks everybody.

Maia Samboy 1:01:31

Thanks for coming.