https://users.cs.utah.edu/~aburtsev/5460/hw/hw1-shell/hw1-shell.html ◉───╡ Burtsev's HW #1 https://brennan.io/2015/01/16/write-a-shell-in-c/ ◉───╡ Brennan's Blog about C Unix Shell https://youtube.com/playlist?list=PLfqABt5AS4FkW5mOn2Tn9ZZLLDwA3kZUY ◉───╡ CodeVault's Unix Processes Playlist (for pipes)
So you want to build a shell? Maybe just learn how some of the basic features work? You may be in luck, this post summarizes my experience completing Anton Burtsev's cs5460 HW#1 Assignment where you are to build a Unix shell from scratch using built-in library functions like fork(), exec(), dup(), pipe(), write(), read(), close(), etc using the C programming language.
I am not taking this course officially however am trying to learn from the material for the purposes of gaining a better understanding of xv6 (specifically memory management), and Linux by association so that I am able to roughly explain how Linux memory management works for the Volatility section of the digital forensics course.
This post will outline the different critical sections of the shell (Argument parsing, I/O redirection, built-in shell commands and pipes).
My general procedure for completing this assignment and other programming assignments is I try to use manual pages as much as possible to understand the different function declarations and arguments. If I have to deal with a problem that I can not easily fix, I will attempt to solve it for around 20 minutes before moving to web searches, then repeating the process for varying levels of help until I can both achieve a solution and understand the general way the problem should be solved
these levels of help are arranged in the following order:
1: manuals (20 minutes) - (read examples, learn concepts) - personally acceptable
2: web searches (20 minutes) - (examples) personally acceptable
3: videos (10 minutes) - (examples) personally acceptable
4: AI (5 minutes) - After using it, I feel a really bad mix of 'I don't understand what's happening anymore but I know it works' and 'how do I not reach this point again?' The work doesn't feel like mine and I will often go back again and try to make it again myself. - Hopefully don't get to this point.
I will not explain anything in too much detail for this section as the Utah homework site provides a template which already handles the command arguments, so this was not written from the ground up by me, I can still explain it though.
In the simplest explanation I think I can give, in the sh_loop() function, a while loop is created with will read the line of user input using the sh_read_line() function and save it to a variable 'line', pass 'line' as an argument to sh_split_line(line), and then execute a command based on that output using sh_execute().
sh_read_line() simply uses the built-in getline() function to get user input and handles errors accordingly with fprintf(), but perror()can also be used here.
sh_split_line() is a little more complex of a function and will start by setting 64 bytes to be allocated to **tokens using malloc(), if there are no tokens, then it will handle the error accordingly, to parse the tokens, SH_TOK_DELIM specifies what characters should be counted as delimiters (what actually split a string). This is used in the built-in strtok() function where the output is saved to token. A while loop then begins which increments by 1 for each token that exists and will allocate another 64 bytes of size if the amount of tokens in the input exceeds the size of the buffer where tokens are stored. (clever trick! This has a use in a lot of other programs!). These tokens are then ultimately returned and fed to the sh_execute() function assuming it was implemented by the programmer.
With this, it should be generally easy to understand what the basis of the shell is:
1: Take user input 2: split it into tokens 3: do checks on the tokens to see if any match certain criteria 4: execute unique operations based on matches
I will now move to the parts of the assignment that are not given by immediate examples, I will share code, but it will be in pieces and not immediately functional. I've never talked to Anton but I'm assuming based on the lectures I've watched so far and what he's said, that he would be fine with it as long as it's not the entire sh.c file. I will honor that.
If you've used the shell template you quickly will realize that when you type something like "ls", it's not like your friendly neighborhood bash. No, it's more like a trained interrogation suspect, because this thing isn't talking!
It's pretty simple to understand why this is the case. Your shell is a program, this program doesn't have the ability to read disk or file system entries. ls however, is able to do this, but ls is a standalone program located at /bin/ls and our shell doesn't know this program exists or how to execute it. Sadly, it's not as simple as writing an "exec(args[0])" and being on your way. If you were to try something like this, you may get it working to some degree, but the ls program would take over the shell program and not return to it after ls has concluded it's operation.
Anton explains this in his lectures pretty clearly and even the assignment tells you what you need. You need fork() and exec()! Upon reading the manual for fork(), it's description says it all: "create a child process". I programmed this section in the following way:
1: if there are no arguments; return 1 2: go intosh_launch()function; 3: insidesh_launch(), runfork()and save it to apid_tvariable "pid" 4: insidesh_launch(), returnpid; 5: if thatpidis 0 (meaning it's the child process), then executeargs[0]6: wait for execution of child process to stop, and finally return.
if you are confused by my half-baked pseudo-code explanation of this concept, Brennan has code there that cs5460 is based upon. Anton says don't do it, but it's up to you.
I'd also like to note, that if you just use exec(), you will have to type the full path to the program for it to execute it. The homework assignment says to use either execlp(), execvp(), or execvpe(). If you want to know, I used execvp(), it requires the same arguments as exec() and searches the path, no serious changes other than adding "vp" to your exec() command should be needed if I'm remembering right.
Scary scary! Something not able to be copied from a working solution! Let's take a look at I/O redirection and I'll try to document the general steps I followed to succeed at this.
The Homework assignment has a very useful sentence for this section:
"You might find the man pages for "open" and "close" useful." - King Anton.
For this section, I used open(), close(), and dup2() to get this done. So what was my steps of operation?
First, I had to figure out if my shell could do something other than just execute what I told it to, "how would I get it to recognize the '<' and '>' symbols?" I thought. There are different ways to do this, but I thought putting the logic inside of the sh_execute() function would be the easiest way, as you have access to char **args directly. You could do something like this:
NOTE: This is pseudo-code very close to real code.
int sh_execute(char **args){
//--- general variables ---
// define an int for input and output status
// define a char* for input_file and output_file
while (args[i] != NULL){ // run through each argument
if (strcmp(args[i], "<YOUR_SYMBOL ('>' or '<')>")){
output_file = args[i + 1]; // whatever the next token is = output file
output_status = 1;
args[i] = NULL // remove the '>' so that it doesn't get interpreted as a flag in another program
} else if (you get the point, second symbol here){
printf("Bleh!");
}
i++;
}
}
To understand how I got to this final product, I first checked if I could read certain characters from the tokens and print them to the screen, so I implemented that first. Then I removed the print statement what I could see it was working, then I went back to bash and stared at the syntax of input redirection for a while:
coolbash$ ls -a > ls.txt
When you stare at it for long enough and think, you'll realize that in the majority of cases (I don't think there's a case where it's not like this) The important file you need to send stuff to or read stuff from, is going to be on the right of the redirection character. I decided to go with the simplest approach and just make another variable that stored the data inside the args[] array at i + 1 (one to the right!). That will be useful for when we need to open a file for reading or writing later. Then at the bottom I increment i. You could use a for loop here and it would be cleaner, but I like my while loops.
moving lower, another if statement is put to make sure there are no empty arguments, and finally save the results of fork() from inside the sh_launch() function to the child process, some of this overlaps with the program execution section:
--- child_exec int variable ---
if (args[0] == NULL){
return 1;
}
child_exec = sh_launch(args); // make that sweet sweet fork
if (child_exec == 0){ // if the child_exec var is zero, that means it's a child process!
if (input_status == 1){ // if a read from a file is happening (cat < file.txt)
int fd = open(input_file, O_RDONLY, 0644); // get file descriptor for reading and give it permissions described in open manpage.
if (fd < 0){ // if open fails, stop program.
perror("open()");
exit(EXIT_FAILURE);
}
dup2(fd, STDIN_FILENO);// duplicate the file descriptor 'fd', make the new file descriptor "STDIN_FILENO", which equates to '0'.
close(fd);
}
}
if (output_status == 1){
printf("bleh bleh pleh do stuffs");
}
// the only real difference with the second if statement would be changing the flags around inside of the open function, the open manpage describes these flags and what they do. Something the manpage or assignment doesn't explain well is that you can have multiple different arguments inside of one parameter position to do multiple different things at the same time if you use '|' characters to split them. It's like this:
open(output_file, O_EXAMPLEFLAG1 | O_EXAMPLEFLAG2 | O_EXAMPLEFLAG3, 0644); // those pesky parameters!
// this technique is useful because is allows you to do multiple things like make the file write-only with O_WRONLY, but also if the file doesn't exist, create it with O_CREAT.
This is the basis for IO redirection and from my experience, is so much easier to understand than pipes. I will continue to hate to implement pipes until I understand them.
Before we get to that pain though, I did end up implementing cd on my own completely (the rush I got when it worked was amazing, it was like I won the lottery.) So I'll show that off, if you don't want to hear how I did it, Again, Brennan shows it off, and he's got a lot more functionality than I do!
I implemented cd in a sort of weird way, but it's still in the context of this assignment better than Brennan's in my opinion, because it's made using the template functions and is a lot easier to implement from the templates than his.
I modified the sh_launch() function to first run a strcmp() on the first argument to see if it was "cd", if it was, I returned 2. Then inside the sh_execute() function, I made another if statement, if the return value from sh_launch() was 2, change the directory to whatever is in argument 2, if it's not there, throw an error:
int sh_execute(char **args){
--- previous stuffs ---
if (output_status == 1){
printf("bleh bleh");
} else if (child_exec == 2) {
if chdir(args[1] != 0) // changes directory, if it isn't zero, then throw an error
perror("cd")
}
}
int sh_launch(char **args){
if (strcmp(args[0], "cd") == 0) // check if any tokens equate to "cd".
return 2;
else {
// fork logic goes in here
}
}
And that's the cd implementation, very simple. Now I don't need to run back to bash like a lost puppy when you need to change directories!
And now on to the final boss of my coding playthrough, those evil pipes. This was the one thing I needed AI to help me write (This is probably pathetic to at least one person reading this). I still don't fully understand this and find the manpage example for this kind of hard to translate into a shell setting. So now I will try and tackle this once and for all!
From reading the AI code and looking at some additional youtube videos which cover the general pipe procedure (I like CodeVault's Unix playlist for this, just look at the pipe videos). I believe the general procedure is as follows:
NOTE: this is only for a single pipe implementation, but multiple pipes can be achieved by slightly modifying this procedure.
Write pipe:
1: create variables for the commands to the left and right of the pipe 2: make the pipe usingpipe()3: check for pipe errors after spawningpipe()4:fork()the first child (remember to make a new variable forpidwithpid_t) alsoclose()read end of the pipe before doing anything else. 5: follow same error checking as #3 but forfork()6: write to the pipe as child #1 by changing file descriptor usingdup2()and close the write end of the pipe. 7: handle redirection if it exists in the string 8: execute the first command 9: check for errors Read pipe:
1: follow same implementation steps forfork()as write pipe 2: make child #2 read from the pipe, make sure to close the write end of the pipe before anything else 3: change the file descriptor usingdup2()to read from the pipe, then close the read end 4: handle redirection the same way as write pipe 5: execute the second child command
It's like everything before the execvp() function is just pipe staging to set up the pipe, copy the file descriptors, figure out which file descriptor is reading and which one is writing, and then after all that, you finally execute.
As I said before, I sadly used AI for this part, so I won't share any code from that (I'm ashamed!). I will publish this now, but this page may be updated with pseudo-code that I completely understand and think you will be able to understand too in the future when I get the hang of pipes.
First and foremost, I'd love to give a huge thanks to Anton, Brennan, the MIT people who influenced the creation of this course, and any past /present/future TA's that help with the material for this course. Being able to learn this stuff for free is awesome.
I will likely make more blog posts into the future as I continue to work through this course and try to work with xv6, so more C bits in the future.
Also, if you're waiting on that dd forensics section, it's done but I'm just making sure it's decently easy to comprehend and understand, so that should be out probably within the week (hopefully).
Until next time!