Secure by Design: DevOps Security for IBM i

Dan Magid         00:05

Great. Okay, so in the description it talks about, we’re going to do a lot of demos here. So I’m going to start out with some PowerPoint slides and then we’ll actually get in and take a look at what does it look like to set up this kind of secure environment for your IBM I code. And so the question becomes, you know, why are we talking about DevOps as part of a security strategy? You know, why are we worried about that? And if you look at our, this is our security maturity model for IBM I customers and the kinds of things that you can do. And if you’ve been in our other sessions in this webinar series, you’ve seen this where you’ve got the sort of basic authentication at the bottom here and then moving up into TLS and encryption and then things like multi factor authentication, token based security and authentication. And then here at the top we’ve got the monitoring. Today we’re going to be focusing on level four here. That is securing your DevOps environment to make sure that you know how every change everything that happens to your source code into the objects that you’re running, how they get to where they are. So exactly how are changes made, who makes the change, when they make the change, why they made the change and then ensuring that you’ve got a secure audit trail for the entire process. So that’s what we’re going to be focusing on today. So this is the question I always like to ask IBM I people, do you know what’s actually running in your production environment? You know, we have a lot of people who have opinions about what’s running in their production environment. But I have for now been doing change management on the ibmi. You know, those of you know my history, I was with Alden for many, many years. So I’ve been doing this for about 35 years. And I almost never go into an environment where when we run our initial scans that we find that the objects and the sources are where everybody thinks they are. That, that, that you have, you know, your pointers in the objects that say, here’s the source. This is where the sources that I was created from. And you go look there and it’s not there. The source was created in somebody’s development environment and then it was copied from there to somewhere else. So very, very few companies, I have found, even companies using things like Alden, like turnover, like MDC ms, the different change management systems, even customers using those things often find Surprises when they go in and take a look to see do I actually know what’s running in production. So one of the things we want to make sure is that you always do know that, that you always know exactly what source the objects you’re running were created from and where that source is at any moment in time. Make sure that the creation dates. One of the things you can do on the IBM is you can actually manipulate those creation date and timestamps so you can actually change them. You want to make sure that that’s not happening. You need to know when you did the build what library list was used. Because a way of getting into code would be if you wanted to change somebody’s code, you could say, you know what, I’m going to slip a library into the library list that has one of the copy members. A copy that I’ve create in that library list. When I compile, it’s going to use my code, not the code that we’re supposed to be using. So you want to make sure you know what that library list is. And then are you actually deploying the objects that you created from your process? So ensuring that you know what’s in production means you have control over all of those levers. And then is your production code actually secure? Who has access to it? I have found many, many IBM I sites where the developers have all object authority, they have SEC offer authority, they can get into anything and they can make changes. And sometimes they do, and sometimes they’ll have good reason to. They’ll say, we had an emergency, I had to go in and make a fix and I just went right into production and made the changes. That’s something you probably want to make sure you have a different process for how you actually make those changes because you don’t want to allow for that corruption. I remember I was talking to an IT director who once told me, he said, we have a very strict move to production process. Everything has to go from development to test, to QA to production. We enforce it all the time. You always have to do that unless the programmers are in a hurry. I said, well, have you ever met a programmer who’s not in a hurry to get a change into production? So you want to make sure that you have an enforced process for the regular things you’re doing and even an enforced process for if you have to make that kind of an emergency change, then the other thing that’s really important is where is your source of truth? Where is it that you have the copies of all the source code that Actually is what you think production was created from. You want to have that in a place that’s secure, that cannot be manipulated. You want to know that that’s where the source of truth is. And we’ll talk a little bit about where you can have that source. So you have an uncorruptible source of truth for where your source code is being stored. And then do you regularly back up and replicate production source code? One of the things we found with some of our customers is their replication software doesn’t replicate their code libraries. It replicates all their database, all their tables and views and whatever, but it doesn’t replicate their actual source code. So you want to make sure that that stuff is getting backed up. Do you have some sort of a review process as you move changes through? Is somebody taking a look at those changes, or is it just, you know, the developer makes the change and moves things forward? So is there some sort of a review process before you move things through and you have the separation of duties to say there’s a request process before an actual promote process into production, where there are two different people involved in moving changes into production, and then are you logging everything that’s happening and just some of the things we know that have happened. So you know, the very famous Target retail hack that happened a few years back was a DevOps problem. Some malware was deployed through their system, but what happened is that the hackers compromised the credentials of one of their business partners, is able to get into Target system and deploy out to all of their POS systems, and malware that was recording all of the credit card numbers and sending them out to the hackers. So it was in their deployment process where they had this problem where they didn’t have good control over what was actually getting sent out to their POS system. So they didn’t have the right controls over that. Again, hugely expensive. 70 million customers were affected. Cost them $200 million. They think this is a common one. This is one you see popping up all the time. The first time I saw a record of this is in a book called Computer Capers that came out in 1979. This is what they call the rounding hack, where in interest calculations or wherever you’re calculating monetary transactions, where there are fractional cents, usually you want to round to the nearest center, the nearest whole cent. But what the hack does is they put code in that instead of rounding, it truncates the extra decimal positions from those numbers and simply accumulates all of those missing fractional cents into an account that Goes then to the hackers when you have millions and millions of transactions occurring that actually can add up to lots and lots and lots of money. This hack has actually come up multiple times in history. Like I said, first time I saw it, 1979. But it’s several times since then where companies have been hit for literally millions of dollars. And it’s again not having good control over the code that somebody was able to put that code, that malicious code and promote it into production. This is actually one that I personally knew about when I was working for IBM in Nevada. One of our, one of our casino customers had an IT director who suddenly disappeared. And what they found was that he had embezzled millions of dollars from the casino and buried what he was doing in the code. So in the, in the casino accounting code, he had put in things that allowed him to skim money and send it to himself. Now he disappeared, so we don’t really ever know what happened to him. But anyway, he sold millions of dollars from the casinos. So lots and lots of things can lead to these kinds of problems, you know. And again, I could come up with examples of many, many, many examples of things that have happened. But again, the way those things happen is the programmers have all object authority or they have SEC offer authority so they can make whatever changes they want into production. Or too many people actually have access to wave promote to production process. So anybody can do those promotions. And maybe they’re being done by a generic user ID for doing the promotions into production. So you don’t have control over that, you’re not scanning, nobody’s reviewing the actual code changes. So maybe code changes are getting in there that you don’t like. And an operations person simply pressing a button to move that code into production, they’re not actually reviewing the code that’s coming. There are no scans being done, no automated scans of the code to look for things like security vulnerabilities and very, very little audit trail of the actual changes that are being made. So there are lots and lots of things in the IBM I world that can create these vulnerabilities. And one of the bottom lines is that we rely in the IBM I world very much on trust. And that is, you know, we have developers, and those developers over the decades have proven to be very trustworthy people. You know, we have people who have been working on our applications for decades and if they wanted to steal from us, they would have done it a long. So we basically trust everybody. The problem is in today’s world, if their credentials get compromised and the vast majority today of hacks of people’s systems are because of compromised credentials. Where a hacker gets a hold of credentials, even if you trust the developer, if somebody gets a hold of their credentials and they have these high levels of authority, that hacker can now do the things that we’ve talked about. They can make changes to your code. And I know Aaron talked about this. Aaron said that if he was a hacker, he would go into the IBM I, he’d find some object that had a last change date that was really old, so he knows nobody ever actually looks at that thing. And he’d go into that program and he changed that program and then reset the last change date to what it was originally. So it looks like it hasn’t been changed in years and it could be sitting there and running on your system and you would never know it. Right.

Aaron Magid     10:48

And Dan, actually, if I can, if I can add into that. Yeah, that is exactly the kind of thing that we talk about over here. And these kinds of changes get especially powerful as IBM adds more and more power into the languages. Right. One of the things that we’ve looked at is now that you have access to these stored procedures in the database that basically allow your programs to do just about anything. They have really, really powerful capabilities in there. Means that if I can get that code in there, I can be triggering network operations. I can be doing just about whatever I need to do on the system in order to get in. And you couple that with Dan, you’ve been talking about compromised credentials. I don’t have an actual stat on what percentage of IBM I developers do this, but it seems like it’s every meeting that I’m on most of the time when I’m on a meeting with a team, when we need to log into the IBM I, at some point, the developers I’m on with will pull up a notepad document or a Word document, and inside of that document they have a section that says, okay, here’s my developer user profile and password. And then here’s my SEC offer user profile and password. Right. And so what that creates, you know, though I understand why people do it, what it creates is a very, very easy hole with which to exploit those credentials. Right. Those credentials can be leaked extremely easily. And at that point, again, you can get in, you change some code, deploy it out, and everybody won’t even know. There’s no evidence that that object has actually been changed and no way to validate that the object hasn’t been changed.

Dan Magid         12:29

Right. So again, so the idea is we want to make sure we have control over that. So there are many things we need to do. So we need to make sure we’ve controlled the access to the code. So who has access to the code in that we have somewhere where we’re keeping that source of truth, that we make every change and associated with some kind of a task. And this is, you know, a lot of people don’t do this. This is important because if you’re going to then implement an approval or review process that gives context to the reviewer of what is it you’re trying to do. So when they look at the code, they can say, okay, yeah, what they did here makes sense given that this is the task that they’re trying to perform. You want to be able to say, here’s what it is that I’m doing when I’m making this change. You want to make sure that they can’t just get in and change, create options or change the ownership of things as they’re moving into the production environment. You want to make sure, as I talked about before, that you control that build library list, that you have some sort of a request and approval process, at least for production, so that you know that there’s an appropriate separation of duties. And the same thing if you’re going to be deploying to other machines. This one of my favorite stories was I was working with a very large pharmaceutical company and they had 35 IBM eyes at their different plant locations around the world. And they had one central development environment and it was split into two machines that had the development machine and then there was the machine that actually did the deployments out to the 35 actual production machines. And the IT director, who was actually not an IBM I person, but the IT director, I went there for a tour and he was showing me the the room where the IBMi that did this distribution was. And he said, look, it’s in the secure room. Only two people actually have the code to get into this room. We have this machine completely locked down so that nobody can get to the machine. And I said, well, but how does the code actually get onto that machine from the development environment? He said, well, the developers move it. I said, well, what authorities do the developers have on that machine? So they have secoffer authority. I said, that’s great that you got the machine under lock and key, but if I’m a developer, I can move anything onto that machine, which will then be deployed to all the machines in your network. So you want to make sure you have control over those processes and the kind of Documentation you want to have, you want to know what were the tasks that people were working on, what approvals have they gone through, what are the changes that are actually made? You know, who, what, why, when and where. You know what, why did people make this change, what were they doing, when did they make it, who signed off on each of the operations, each of the promotion operations, what was the build process that was used and what was the deployment process that was used. So if an auditor comes in, they can very easily go in and see, well, here’s exactly what you did. So I have absolute control over everything that’s happening in my production environment. So how do you sort of get there? How do you get to that environment? And that’s really what modern DevOps is all about. The modern DevOps environment is to provide that kind of an environment where everything is under control so you can keep control over exactly what’s happening. So, for example, you make your source of truth instead of some libraries and source members on your IBM I, you make your source of truth your git repository in the cloud and that can be completely secured. You can have multi factor authentication on it. You can decide who is allowed to get at it, what is it that they’re allowed to do, so that your code is always in that secure location and that becomes the source of truth. If you look at it, you say, this is the process. I can say, okay, I’m going to create a ticket says here’s what I’m going to do. Then in my code repository I can create a way of isolating the changes I’m making for that ticket. And that in a git environment might be by creating a branch, you can say, I’m going to create a branch for this particular task I’m working on. And that way all the changes I make for that task are associated with that branch. That way I can even make changes for another task to the same programs. And yet I still can see which change was for task one, which changes were for Task Task too. So I make those changes and then once I commit them to my repository, I can then move those changes through some kind of defined life cycle. So I know for every change what’s the purpose of the change, I can see what’s currently active, what are people doing with those particular tasks, and then I have an automated process to move things through. So as I move forward, then I can move those changes to a shared repository where other developers can have access to them. I can make my own changes in isolation in my own library. And then when I’m ready. I can share them with other people so that they can work with those changes. Then I can move them through some life cycle where maybe I go to an integration test environment to test multiple tasks together to see how they work. And then I can go to say, a QA environment, a quality assurance environment and see how things are, whether we pass our quality tests. Basically, I can move through that entire process before I finally get to that final move of the code to production. Let’s actually take a look now at how does that process work? What does that process look like? What we’re going to do is we’re going to work in an environment where I have my code and my libraries on my IBM I and I’m going to make changes on my IBMI and then I have my local repository for, for tracking the things that I’m doing in my own development environment and for, for storing the things that I’m doing in my own development environment. And then I’m going to have the shared cloud environment where I’m moving everything once I’m done with them. And that again is going to become my source of truth. The source of truth becomes the git repository, not what’s in my libraries. Because again, on the IBM I, if I have enough authority, I could go in and I could make some changes into those libraries that are not being tracked by the change management process. If it’s in the git repository, if that’s my source of truth. The only way to get in to make those changes is getting into that git repository. And everything I do in the git repository is going to get tracked. Everything that happens, it’ll know who did it, why they did it, when they did it. All of that is automatically tracked. There’s no way to change that without going through that git process. So that becomes your source of truth, of where everything is. So that if something should change, you know, I can I, that I, I will know about it. So let’s take a look. I’m going to flip over here to, to, to my, my cloud environment here. So here’s my, here’s my cloud repository and here you can see I’ve got a bunch of stuff looks like IBM I source files. And that’s really what they are. But if for git to understand them they have to be in directory structure. So one of the things that Eridani does is we actually get your code from the library source files and source members into a directory structure that Git can work with. Now that is also important because we’re going to see in a minute if you want to use tools like Sonar Cube or Black Duck for doing source scanning, or you want to tie it into tools like, like Jira or ServiceNow or ClickUp or other kinds of issue tracking systems if they’re in a git repository. Those tools all work with things that are in git repository. They do not necessarily work with things that are in libraries and source files and source members. Once you have things in a git repository, suddenly the whole world of all these other tools opens up to you. You can use all of these other tools. Now, here you can see I’ve got my environment set up here and I’ve got my base production environment. I’ve got a quality assurance environment, I’ve got an integration environment, and I’ve got a task environment. So I’ve already created a task here for me to work on, a task environment for me to work on so I can make my changes in association with the task. If I wanted to, I could create another, another task that I could work on. But basically I can make all of my changes here. So what I’m going to do is I’m going to flip over now to my green screen and I can go in right here and make changes. But first I’m going to do a get status which will tell me what’s happening here. And it tells me, oh, you’re working on task one. Okay, so I’m working on task one. That’s the. Whenever I make changes, it’ll be working there. Now, if I wanted to, I have the ability to say, you know what? No, actually I want to work on a different one. So I could say, you know, I want to work on task two. And if I do this, it will actually change my library to reflect the things that are in task two rather than in task one. So I can at any time decide which task or what branch I’m actually making my changes on. Now, git may or may not allow me to actually push changes into that branch that I can control as to whether or not I can actually push my changes into that particular place. But anyway, right now, simple. I just want to go in and start making some changes and work the way that I normally work. So again, I can work here in PDM by going into workmember PDM and making changes. So let’s go in here and I’m going to make a, let’s make a change to this, to this program one here and we’ll change so we can see this. We’ll change this to the 29th. We’ll make these the 29th. All right, so I’ve made that change now in here. Now I can also go over here into Visual Studio code and we’ll go in Visual Studio code and we’ll come in and we’ll make a change to this program too. So I can make changes in Visual Studio code and we’ll do the same thing. We’ll make this 4:29

Dan Magid         22:39

same thing with this one. I’ll make my changes in Visual Studio code so I can do that.

Dan Magid         22:50

Finally, if I want to, I can go into RDI

Dan Magid         23:02

and I can make my changes in RDI so I can make the changes in whichever one of these environments that I want. So again, everything is still going to be controlled with the same process. I’m going to know exactly who makes the changes and what changes they make, regardless of which of those IDEs they’re using. So it doesn’t matter to me. And I as a developer, I can work in each of the different ones if I want to, or I can just work in one. I can have different people working in different ones of the IDs. All of that, it’s completely up to me is to exactly how this process works.

Dan Magid         23:42

So I’m going to go ahead and make a change here. So let’s see, let’s go this time. Let’s go. And I’m going to go and make, let’s make a change to a physical file

Dan Magid         24:01

and we’ll make that 429. Let’s go ahead and save that

Dan Magid         24:11

and we’ll change this physical file.

Dan Magid         24:24

All right, so I’ve made my changes and now if I go back over here to my green screen for a second

Dan Magid         24:35

and I’m going to go in and look at my, my, my development object library. So where I put the objects that I’m actually going to test with, I’m going to go in my Cudev1obj library. Let’s see. I don’t have anything, so I have nothing there. Oh, and by the way, before I do this, let’s go ahead and I’m going to go ahead and do a git add. So I’m going to tell git to go ahead and add those things and it’s going to go and find all the things I changed. So if I go in now and look at my git status, even though I change some things in pdm, I changed some things in VS code. If I go in here now and look at my git status, it tells me it knows here are the four things you change. So you change these four things. Now what I want to do is I actually want to go ahead and build them so I can work with them. So if I go here to my green screen and I do a work live cudev1obj my object library, there’s nothing in there. So right, there’s nothing in that library. Right now what I’m going to do is come back over here into RDI and again, I could do this from rdi, I can do this from VS code, I can do it from the green screen. I’m going to run the build process. I’m going to say go ahead and build this stuff and I’m going to say I want to have it actually build for all of my development libraries. Now in our case we only have one, but I could have as many as I want.

Dan Magid         26:13

I’m going to go ahead and run that build process now. It’s actually going to go out and it’s going to build all of the objects that were changed and everything that uses the objects that are changed. It’s going to build everything for me, all the things that I need in order to actually test this change. So we’ll give that a second to run

Dan Magid         26:37

again. By the way, it’s automatically setting the build library list so I don’t have to set that, nor do I have to worry that somebody can change it. It’s setting that automatically. It’s also setting all the create options automatically. So this process is controlled. I know exactly how this process is happening. So if I go back over here now and I do my work live again of my CU D V1 obj library, I now have everything in it. So it’s now built all the things I need in order to run my tests and I can go in and run my tests and let’s just say that everything worked fine, I’m happy with the tests. I can come back over here into RDI and I can say, okay, let’s go ahead and commit those changes now. This is an important part of securing the environment is that when I do a commit, I always have to say what it is that I’m doing, why am I doing this? And again, that is important for the person who is then going to have to review the changes so they understand what it is I’m trying to do. So I updated the fields in the inventory file for new warehouse functions, whatever. And if you want to a git convention is I can put an issue number so I can actually tie it to A more descriptive issue, you know, let’s say like a, like a JIRA ticket number or a ServiceNow ticket number. So I can tie it to that as well and I can commit those changes, right? So now if I go in again and do my git status here,

Dan Magid         28:22

it no longer says I’ve got those four modified objects or sources. It now says I have one commit. Now what I’ve done is I’ve committed it to my local development repository. I have not yet shared this with everybody else, So I now have this change in my local environment, but nobody else has access, nobody else has access to the changes that I’ve made. So now what I can do is I say, okay, let’s say I want to make some more changes. So I committed the changes. And then I went, oh, you know what, I should have done something else. I should have made some other change. So I can come over here, back into my library and this time we’ll just work through PDM and actually, let’s go ahead and change, just so you can see this, I’m going to go ahead and change,

Dan Magid         29:14

go ahead and change the physical file again. So I’m going to go ahead and make another change here to the physical file. So I’m going to make

Dan Magid         29:26

this date also. I’ll make this date also the 29th, and I’ll make this one the 29th and this one the 29th. All right, so I’ve made some more changes.

Dan Magid         29:44

I do my git add like we did before, and then if I do my git status now, what I’m going to see is, okay, you’ve got this modified physical file. You still, you’re still ahead by one commit, but you’ve now got this modified physical file out there. So it’s, it’s something else that’s changed. So what I can do now is I can do another commit. So I can go in and say, okay, well, let’s go ahead and commit that as well. So I’m going to do another commit.

Dan Magid         30:16

I fixed a bug in my last update, so whatever it is, I want to put in here, same thing, I can put in the issue number if I want to,

Dan Magid         30:35

I can commit that. And if I do my git status again now, it’s going to show me now I’m ahead of the shared repository by 2 commits. It actually is tracking those changes separately. So I can see what did I do with each one of those commits. Now I’ve made those changes I actually have now in my repository. If something was to happen to my development Environment, I could restore things directly from the repository and rebuild my development environment so I could get those changes back. The other cool thing about how Git works is let’s say somebody says, hey, I know you’ve been working on this big project on task one, but we have a production problem and I need you to get in there and fix it. Right now. I actually could create a new branch called emergency, switch to that branch and then my library would reflect then the environment. Without the changes I made for task one, I would actually be reverted back in my library to the environment before the changes for task one. I could make those changes, push them up, and then switch back to task one and see all the changes again that I had for task one. I have the ability to work on multiple things again without losing that traceability of who, what, when, why for every change that I make. So basically now I’ve made some changes I’m working on task one. I’m going to say, go ahead and push these up to the shared repository. So I’m going to go ahead and push these up to the cloud. So no longer just working in my own development environment, I’m now working in the shared environment. And if I go over to my cloud environment, right here’s where I was before on task one, you know, last change was 26 minutes ago. If I refresh the screen now,

Dan Magid         32:28

I’m going to say, oh, okay, wait a minute, now I’ve got those two, those two commits, those two changes. And again, if I was to go into my history,

Dan Magid         32:39

I can see those changes right here. Here are the changes. I can see the information. There could be descriptive information. I can see all the information. And I can also go directly in here and say, oh well, show me what’s in those changes. And I can actually see, oh well, you change this physical file, this physical file, you change this RPG program. I can see all the changes at any time. I can see that information. So I always have visibility into the changes that are being made. There’s no longer any changes that are being made that I can’t actually look at that I can actually see what are the changes that are being made. So then let’s say, okay, now what I want to do is I want to move, I’ve looked at task one. I want to move everything from task one up to the next stage of my life cycle, which in this case is the integration environment. So I now want to move things up into integration from task one. If I was to change my view and say, let me look at the Integration environment. So now I’m looking at that. I can see those changes that I just made. They’re not here yet. I don’t have those changes here. But what I might want to do is before I do that, I might want to say, well, wait a minute, I want to see everything. If I’m about to take everything from task one and move it up in integration, maybe I want to see everything that’s been done. What are all the things that are in task one that are not yet in integration? So I can say, okay, well, take task one and compare it to integration

Dan Magid         34:16

and show me what changes have been made. Here you can see those changes I just made and you can see the actual changes. If I’m the reviewer, if I’m assigned to reviews, and we’ll see in a second how you can actually require that to happen. I can see every change that’s been made and I can see them in the context of. Of each of the commits. So I can look and say, well, here’s what they were trying to do. I want to actually see, well, what did they do? And I can go in and say, okay, updated the inventory for the inventory file. Here are the changes that I made to do that so I can actually see the context and see the code to have an appropriate review process to make sure you know what changes are getting in production, why you have the context and the actual list of changes that you can look at to see exactly what was changed and why those changes were made. Once I’ve done that, I can say, okay, I want to go ahead now and merge. I want to take all that stuff that was sitting out in my. I want to see everything that was sitting out in my task one environment, and I want to merge them up into the integration environment. I’m going to create a merge request again so you can separate the process into a request and a fulfillment of the request. Here you can see it automatically said, well, if you’re moving from task one, the only place you can move is integration. It did not let me select where this is going to go. It’s actually saying, this is where it goes. And you can decide. Is somebody authorized to say, well, wait, this is an emergency. I want to move it right to production. You can decide if they have the ability to do that or not, but you can say, nope, this is a fixed process. The only way you’re getting this stuff up to the net to move forward is it has to go to integration so I can make that process happen. And again, I can see here exactly what I’m about to merge. And if I wanted to, I could do another review, go back and look at the, at the actual changes. But I’ll go ahead and create that merge request to say I want to move these things forward. And again, you can put in all kinds of things. Now notice it says here, for this stage, approval is optional. So I don’t actually have to get an approval for this for a move to integration, it’s optional, I could get it, but I don’t have to. So I can just go ahead and say, yeah, go ahead and merge it again. I have another opportunity there to look at the changes if I want to do that now it’s gone ahead and it’s done that. And by the way, if I want to, I can go in here and actually see what are all the things that this guy Daniel Maggot has done, you know, and when did he do them. So I can see every single activity that this person has done. So if you know somebody was to leave the company or you want to check in on what a contractor has been doing, you can actually see every single activity, you can see everything they’ve done and you can drill into any of these activities and actually see the changes that were made in association with that. So again, complete audit trail and very, very easy to see who’s doing what. So anything that’s being changed you can see. And again, this is in the, in the repository. So that’s why again, it’s really important that the repository is your, so your source of truth. That that is, is, is the actual source of truth. Because everything that’s happening in this repository is being controlled. So if I go back now and look at the repository and I look at my integration environment,

Dan Magid         37:58

I now have those changes here in my integration environment. I’ve now moved those changes up to the integration environment. I now have visibility into that. And again I could go in and look and see, well, what did I just move? And I can see that list of things, I can see everything that has been done. But at this point we’ve moved things in the cloud repository. So they’re in the cloud repository, but they’re not in my IBM. I. If I go over here back to my IBM I and I change my current library to CUITG1 to my integration environment and I’ll do a quick git status and it’ll show me I’m on the integration branch. But if I go into workmember PDM

Dan Magid         38:44

and I go into that integration environment

Dan Magid         38:50

and I take a look at my file here, right? I don’t have those 429 changes here. They’re not here yet. They’re sitting up in the repository, but I don’t have them here. So what I’m going to do is I’m going to do a get poll to say, okay, I want this to come down to this environment and in a minute I’m going to show you how you can automate all this. But what I’m doing now is saying, okay, get me the stuff from my source of truth from the repository and bring it down here to my libraries.

Dan Magid         39:22

So I run the git pull and then I go back into that file and now I have the 429 changes. So now I have those changes here. And just like I did through RDI the last time, I can do a build here.

Dan Magid         39:46

If I go, I’m going to do a work live with my CUITG1OBJ, my object library for integration Notice, I don’t have anything in there. I can go in now and say, okay, let’s go ahead and do my build there. So I can do my build

Dan Magid         40:19

now again, it’s going to build. It’s going to do the process of creating all the objects necessary to test the changes in the integration environment. So again, all the process is controlled. I know exactly what library list is being used for this. I know exactly how the objects are being created. I know what options are being used to create the objects. I know what, what authorities are being set. This is all being done now for me automatically. So I don’t have to worry that somebody is getting into the process and changing how these things operate. It all, it all happens for me, for me automatically when, when I perform these operations. And again, if I, if I had only changed one program, all it would do is create that program. It doesn’t create, doesn’t have to create everything. It can create just the, the things that have changed. We’ll just give that another second.

Dan Magid         41:27

Oh, sorry, I hadn’t hit enter. I hit enter and got a duck more prompts and didn’t hit enter. All right, now it’s actually going through and creating everything. I was wondering why it was taking so long.

Dan Magid         41:43

Aaron, you’re supposed to notice things like that. All right, now if I go in and I look at my integration environment, my object environment, right now, it’s built everything for me. It’s built it for me automatically. Now let’s say though that I don’t want to have to do that. I don’t want to have to go through and do the poll and do the build and do that automatically. I just want to be able to say, all right, we’ve approved this. Move to the next stage of the life cycle automatically do that for me. I have that set up for the QA environment. So if I go in and I do a work live with my Cu qua1obj library, my objects for QA testing, there’s nothing there. And in addition, I actually have another library that I deploy just the objects to. So if I do a work live into my cu qua1 dep, which is my deployment library, you’ll see I’ve got a bunch of SQL stuff in there, but I don’t have any programs. None of my regular program objects or physical files the for the the that we were working with. So what I’m going to do now is go back over here to my, my cloud repository and say, okay, I’m ready now, everything’s okay in integration. I want to go ahead and move these things from integration into QA and same thing actually before I do that, if I wanted to, I could have gone in here and said, you know what, before I do that, let’s go back over here to my code and say before I do that I want to know what’s coming, what am I about to do. So if I want to, I can go in and I could do a compare of the revisions and this time I can put integration over here and say, well, show me what am I going to be moving into QA when I do this. Oh, whoops. Actually now just created the merge request. Let’s go back before I create the merge request and let’s just do that, that compare. So I’m going to compare integration to qua

Dan Magid         43:47

and here I can see, here’s what’s coming. These are the things that are actually going to go into that. That. So I, you know, I may have moved bunches of things in an integration test. I can see the list of everything that’s coming and again I can drill in directly and see all of the detail changes. Actually, one of the things that’s really cool here too. If you use your browser functions, I can go in here and say I want to print this and I want to print it to a PDF and what I’m going to get is this really cool PDF document. I get this PDF document that has all that information on it. So I can actually create a document that says here are the things that we’re committing and these are all the changes that were made. Anyway, I See the things. And I’m okay with that. I say, go ahead and create this merge request and I’m going to go from integration to qa. Again, that’s automatic. It knows what I’m supposed to do.

Dan Magid         44:42

And I could put in detailed descriptions of the things that have to happen on this process and I can go ahead and create that merge request.

Dan Magid         44:54

So I’m now going into the QA environment and notice up it didn’t let me do it because wait a minute, you can’t. Sorry. All required approvals must be given. So if I had a whole list of approvals, that would have to happen as part of this process. Or maybe I say, nope, you have to pass the code scan and we’ll look at that in a second. But say, no, you have to pass the code scan before you can do this, or no, you have to have run your automated unit tests before this can go. So I could have a whole bunch of rules of things that have to happen. So right now I can’t actually move this because I don’t have the approvals I need. So what I’m going to do is I’m going to flip over real quick over to here as a Here is a different user and I’m going to go and look at the merge requests and it’s, oh, here’s a merge request integration to qa.

Dan Magid         45:50

And I can go in now and say, okay, I want to go ahead and approve this request and I could go back in and look at all the commits that were part of this. If I wanted to see all the changes, I could go and look at all the changes, all the code. I can see everything associated with this before I actually do my approval. But we’ll go ahead and approve this. Okay, so now it says, I’m ready to merge. And if I go over to this site, it also says, I’m ready to merge. So it knows that I’ve gotten the approval and I can go ahead and do this. So I’m going to go ahead and run that merge. Say, okay, I want to go from ITG ITG to the qa. But notice now it’s also started a pipeline, it started an automated process. So it said, okay, as part of moving stuff in the repository from integration to qa, I also want to run a series of operations. Again, you could have this be run a source scan, it could be run some automated unit tests, but I can actually go in and take a look at what’s actually happening, what process is running as part of that, I can go in and see what are the pipelines that are. That are actually running as part of. Whoops, let me go back here. What are the pipelines that are actually running as part of running this process?

Dan Magid         47:16

So here I can see. Oh, it’s the deployment pipeline and I can go and take a look at that. And it’s actually already run, but if I go back and look, I can actually see an entire record of everything that’s happened. But here I can go in and see. Okay, well, here. It set my current library. It actually then did the git pull. So it pulled the code down to the ibmi. It then did the build. It did the build. And it, by the way, set the library list for me. I did not have to set that library. It did it. It automatically set the library list for me. Here are the things that it built that shows me exactly what was built. And then it packaged the code up to send it. It then sent it. It then unpackaged it and installed it. So it shows me this. Now this information is going into the history. So at any time, if I need to, I can actually go back and see, well, what exactly happened? What was the library list we used to compile? What, where did the code go? How was it deployed? All that information is being stored. So again, you have this complete history of everything has happened. You don’t have to worry about, gee, maybe somebody changed the setup. Library list. This is one of the issues with things like Alden that we had when I was with Alden is that you could set the library list, but if you changed it, there was no record of the library list having been changed. You don’t know that three weeks ago you created it with a different library than you’re creating it with today. In this, you can actually go into the history and you can see all of this information. So you can see in the history exactly how things were done, what the process was that was used, who approved it, who, who actually did it. All that information is right, right here. Right. I can see everything that’s happened. And I can go. If I want to go back into my, into my code repository

Dan Magid         49:15

and say, I want to look at the, you know, I want to look at the QA branch and I want to look at the, the history of everything that’s occurred.

Dan Magid         49:32

So I can see all the things that have been done. And if I, if I drill into this, that last merge that I did again, I can see all the things that were merged in. So I can see the list of files, everything that was changed. There and if I want to, I can actually drill right into that pipeline and I can see exactly what was done. So I can see all the, you know, all that information is stored. So I know exactly how was it built. I can see all the, you know, these are all the compile listings. I can see everything here that happened in that, that build process. So again, I’ve got this very secure environment where I’m keeping track of exactly what changes are being made, those changes are getting reviewed. I’m able to see everything that has occurred throughout the history of the entire change process. And here again, we got to see it all automated. So all I had to do was do the pull request and approve the pull request. So it auto automatically. It automatically did that process for me.

Dan Magid         50:38

So basically that’s creating that controlled environment for managing the changes. So first of all, I have the secure repository where all of my code is and I can secure that with things like Multi Factor Authentication. So people can’t get into it without doing Multi Factor Authentication. It is tracking everything that occurs. So I know every change that that happens. I don’t have to worry that somebody’s making changes that I don’t know about. Every change is being tracked. I can set up a fixed process that said code goes from development to test to QA to production or whatever I want that to look like. And the code will have to move through that process. It will track who requested the move, who did the move. That information is all there. And at any of those points I can drill in and see. But what changes did they actually approve? I know what reason I was making those changes. All of that stuff is happening. So I have now a very secure environment that I will always know what the source code is that created the objects that are running and I will know how they got there. So if I flip back over here real quick.

Maia Samboy    51:46

Hi, Dan, I just want to say real quickly, we got about seven minutes left if you want to save some time for questions. I know we got one in the Q and A box right now.

Dan Magid         51:54

Okay. Yeah, great. I’m just going to two slides to wrap up. Basically this is the process, right? You’re starting with whatever IDE you want to use, using something like GitLab. I’ve been using GitLab, but you can have BitBucket or Azure, DevOps or GitHub, whatever cloud repository you want to use. At the end of the day, under the covers, it’s all using git. And then the Erdani code will do the builds for you, will do the deployments for you, it’s managing all that and you can integrate it with whatever tools you might want to use. Just last one, last thing is just if you do use something like, like, like sonarqube for doing source scans, here’s the kind of information you see. You get a summary of what it found. It will tell you are there potential injection attacks in your code, are you missing security flags and all of these different kinds of things. It’s looking at your code and saying, hey, you’ve got these vulnerabilities in your code. And here is specifically for RPG code. It will show you for RPG code. In fact, if I click here, it’ll take me right to the sonar and I can see here is a whole bunch of things and I can go in and say, oh, okay, what do they mean by this? They’ll show me. What’s the issue? Why is it an issue? Here’s what you’re doing and here’s the way you should do it. So all of that comes out of these code scans and again, you can have that automated as part of your process that these code scans run automatically.

Dan Magid         53:23

So anyway, protecting your code, you could protect the access to the code. You make sure you know why people are making changes, what are the tasks that they’re working on. You can see all the changes by task. All the build process is tightly controlled, the promotion process is tightly controlled and you get the separation of duties, of requests and fulfillments and then you have all kinds of documentation of exactly why the changes are being made and who’s making them, when they’re making them and why they’re making them. So that’s basically using modern DevOps to secure your environment so you know exactly what code is running in production and how it got there. Okay, so Maya, you said we have some questions.

Maia Samboy    54:05

Yes, we have two questions in the Q&A right now. First, does the git process have controls against multiple developers working on the same IBM? I object at the same time?

Dan Magid         54:17

Great question. And the answer is it has really powerful functions. And you think about that a little bit about the history of Git. Git was, was created for managing the development of the Linux operating system. And so there were developers all over the world working on the same code base simultaneously. And Git had to make sure that when it moved a change from one developer into its main repository that it didn’t lose changes. So yes, it keeps very, very good, tight control over all the differences. And if you make a change and I make a change to the same program, I can move mine into the say up to the shared repository. Let’s say we’re both working on task one. I get my changes into the shared repository. When you try to move yours, it’s going to say you can’t simply move these up because there are changes you don’t have. And then it will give you the ability to merge those changes and if necessary, to actually work with the code to update it to reflect the changes. But it will not allow you to lose changes and it will actually keep track of the change you made and the change I made separately. So it will know exactly what you did and it will actually also track the merge. So it will know exactly what you did when you merged the changes. So you’ll see your change, my change, and then the merge change.

Aaron Magid     55:30

So, Dan, actually, if I can just add one thing to that, and I know you’re touching on this, but I think it’s a really important thing point to make, which is that without these tools, I’m not sure if this is exactly what the question is asking, but it sounds like what we’re asking is do we have a control in place to stop developers from modifying the same object at the same time? And one of the key points that I think is important to make about Git and about these tools is that you don’t have to stop developers from working on the same object at the same time. It’s one of the most important things that Git does that these processes allow us to do. Because Git can manage the deployment of specific change sets, and because it tracks things by change, not by object, we’re able to actually make changes to the same object at the same time with two different developers in their environments, and then promote and merge and work with those changes at a later date.

Dan Magid         56:28

Right.

Aaron Magid     56:28

I’ve worked with git personally for about 15 years, and I don’t think I’ve ever worked in an application that was managed by git where I ever had to say, where I ever had a task to work on an object, to work on a particular source code file, and we had to say, hold on, sorry, you can’t change that right now. You can’t work on your task. We can give you something else because some other developer is working on that right now. We never had to do that. And we also have to go through bending over backwards to make that work because Git made it work for us. I think it’s just a really important point. It will put in those protections to make sure that we don’t get conflicting changes deployed at the same time, that would be a problem. But it also will allow us to work independently so that we actually can work at the same time. We don’t have to throttle our developer teams just because they want to work on the same object.

Dan Magid         57:25

Right, right. The. The analogy I heard somebody once give me. He said, why do cars have brakes? You know, my answer was, so you could stop. He said, no, cars have brakes, so you can go fast. Without brakes, you can’t go fast. That’s a good point. The same thing applies to development here. The. The fact that Git has such powerful compare and merge and protect and features means you don’t have to stop somebody from doing the things they need to do while they wait for somebody else to finish their work. People can just get to work and then you can deal with merging them afterwards. And Git will help you with that.

Dan Magid         58:01

My. Another. You had another one?

Maia Samboy    58:02

Yeah, we have one more. How does it handle dependent objects? For example, when a PF is changed, it needs to recompile all of the programs using it for level check.

Dan Magid         58:12

Right. So. So it knows when you. So that’s actually part of what we call I build, which is our build process. When you first install, it goes out and it does. And this is where we find out that people don’t actually know where their source code is and where the related objects are, because we run these things and people tell us, well, the source code should be here. And we find out, well, now actually it’s not there. We do the scan process where we find all the objects, we find all the dependencies, we find all the create functions. We build a database of all that information so we have all that information so that when we go do a build, you say, well, I’ve changed this physical file. We know to create the logicals or if it’s a table, we know to create the views, we know to create the programs that use those. So we go right up the dependency tree and create just those things that are dependent on the things that you’ve changed. So we’ll go up the dependency and recreate all of those. Same thing goes for things like modules and programs and service programs.

Maia Samboy    59:15

All right, well, that puts us just at about time. If you have any other questions, feel free to email me and I can pass those along to Dan and Aaron. Thank you so much. That was awesome session, Dan and Aaron. And thank you everyone for joining us today. I just really quickly wanted to remind you this is only the third in our series of four security webinars. The next one will be Tuesday, May 13th again at 1pm Central Time. It will be about advanced security monitoring and incident reporting for IBM. I so make sure to join us then I’ll send an email invite as well. And tomorrow keep an eye out for a special blog from Dan that will be breaking down our integration security maturity model that should be coming out tomorrow afternoon. So thank you so much for joining us all and we hope to see you next time.

Dan Magid         1:00:08

Great. Thanks everybody. Thanks Maya.

Maia Samboy    1:00:10

Thanks y’all have a great day.

Aaron Magid     1:00:11

Thank you all.

Dan Magid         1:00:12

Take care. Bye.