See the example program passingArrays.c. This is also available on tuzo as ~jepson/csc270/src/passingArrays.c
I'd prefer if you did use an extra array for b. For example, it would make it easier to solve several different equations, using the same matrix A, but different right hand side vectors b. You don't need to do that in this assignment. You should use an extra array for x, though.
It would be better to check that the element you are dividing by during the elimination is not "too small". This would catch other cases in which the matrix does not have an inverse (Eg a matrix of all 1's... looking for zeroes won't flag such a matrix, but it isn't invertable). Think about what "too small" might mean (Hint: It should be relative to the size of the matrix A in the first place). If, after interchanging rows, your code decides a diagonal element is too close to zero, then output an error message and stop.
See #3 above... You need to do row interchanges and check the size of the new diagonal elements.
Sure. You don't need to do anything special. You need only deal with the case in which the matrix A is invertable. See #3 above.
for example: A b 1 2 3 4 0 1 7 8 0 0 0 1
In #3 above, I said your code should complain whenever a[i][i] is too small. This occurs in your example for i=3 (actually, for i=2 in C. What I mean is the third row and column of A). You are not responsible for trying to find a solution. Your code can complain and simply stop.
No... A is always invertable... always square.
You can. You won't get extra credit though. The only constraint on what programming features you use are: (1) the code must be ANSI compatible C (NOT C++); and (2) the chosen features are suitable for the task.