diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 930a603..7903e69 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -16,7 +16,7 @@ I _don't_ recommend just making a pull request for some new feature—it probabl ## Write a use case -This is probably the most important thing to bear in mind. A great design principle for software libraries is to start with a real-world use case, and try to implement it using the feature you have in mind. _No issues or PRs will be accepted into `script` without an accompanying use case_. And I hold myself to that rule just as much as anybody else. +This is probably the most important part of your issue. A great design principle for software libraries is to start with a real-world use case, and try to implement it using the feature you have in mind. _No issues or PRs will be accepted into `script` without an accompanying use case_. And I hold myself to that rule just as much as anybody else. What do I mean by "use case"? I mean a real problem that you or someone else actually has, that could be solved using the feature. For example, you might think it's a very cool idea to add a `Frobnicate()` method to `script`. Maybe it is, but what's it for? Where would this be used in the real world? Can you give an example of a problem that could be solved by a `script` program using `Frobnicate()`? If so, what would the program look like? @@ -26,13 +26,23 @@ A concrete use case also provides a helpful example program that can be included The final reason is that it's tempting to over-elaborate a design and add all sorts of bells and whistles that nobody actually wants. Simple APIs are best. If you think of an enhancement, but it's not needed for your use case, leave it out. Things can always be enhanced later if necessary. +# Stop + +Stop here until you've raised the issue with a use case, we've discussed it and agreed a solution design, and I've given you the go-ahead to write a PR. + +Once those things have happened, you can get cracking. + +# AI policy + +Use of AI is fine. Disclose it if you like. + # Coding standards A library is easier to use, and easier for contributors to work on, if it has a consistent, unified style, approach, and layout. Here are a few hints on how to make a `script` PR that will be accepted right away. ## Tests -It goes without saying, but I'll say it anyway, that you must provide comprehensive tests for your feature. Code coverage doesn't need to be 100% (that's a waste of time and effort), but it does need to be very good. The [awesome-go](https://github.com/avelino/awesome-go) collection (which `script` is part of) mandates at least 80% coverage, and I'd rather it were 90% or better. +It goes without saying, but I'll say it anyway, that you should provide comprehensive tests for your feature. Code coverage doesn't need to be 100% (that's a waste of time and effort), but it does need to be very good. The [awesome-go](https://github.com/avelino/awesome-go) collection (which `script` is part of) mandates at least 80% coverage, and I'd rather it were 90% or better. Test data should go in the `testdata` directory. If you create a file of data for input to your method, name it `method_name.input.txt`. If you create a 'golden' file (of correct output, to compare with the output from your method) name it `method_name.golden.txt`. This will help keep things organised. @@ -50,13 +60,9 @@ Add lots of test cases; they're cheap. Don't just test the obvious happy-path ca Remember people are using `script` to write mission-critical system administration programs where their data, their privacy, and even their business could be at stake. Now, of course it's up to them to make sure that their programs are safe and correct; library maintainers bear no responsibility for that. But we can at least ensure that the code is as reliable and trustworthy as we can make it. -### Add your method to `doMethodsOnPipe` for stress testing - -One final point: a common source of errors in Go programs is methods being called on zero or nil values. All `script` pipe methods should handle this situation, as well as being called on a valid pipe that just happens to have no contents (such as a newly-created pipe). - -To ensure this, we call every possible method on (in turn) a nil pipe, a zero pipe, and an empty pipe, using the `doMethodsOnPipe` helper function. If you add a new method to `script`, add a call to your method to this helper function, and it will automatically be stress tested. +### Concurrency safety -Methods on a nil, zero, or empty pipe should not necessarily do nothing; that depends on the method semantics. For example, `WriteFile()` on an empty pipe creates the required file, writes nothing to it, and closes it. This is correct behaviour. +Because pipes are concurrent (they may be running multiple commands or filter functions concurrently, for example), we use a mutex to access fields on the `Pipe` struct rather than reading or writing them directly. For example, don't read `p.Env` directly: use `p.environment()` instead. ## Dealing with errors @@ -126,7 +132,6 @@ Here's a handy checklist for making sure your PR will be accepted as quickly as - [ ] Have you opened an issue to discuss the feature and agree its general design? - [ ] Do you have a use case and, ideally, an example program using the feature? - [ ] Do you have tests covering 90%+ of the feature code (and, of course passing) -- [ ] Have you added your method to the `doMethodsOnPipe` stress tests? - [ ] Have you written complete and accurate doc comments? - [ ] Have you updated the README and its table of contents? - [ ] You rock. Thanks a lot. diff --git a/README.md b/README.md index 7e6e332..9386e5e 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,6 @@ import "github.com/bitfield/script" [![Magical gopher logo](img/magic.png)](https://bitfieldconsulting.com/subscribe) -[Subscribe to learn Go with me!](https://bitfieldconsulting.com/subscribe) - # What is `script`? `script` is a Go library for doing the kind of tasks that shell scripts are good at: reading files, executing subprocesses, counting lines, matching strings, and so on. @@ -137,10 +135,10 @@ data, err := script.Do(req).JQ(".[0] | {message: .commit.message, name: .commit. We can also run external programs and get their output: ```go -script.Exec("ping 127.0.0.1").Stdout() +script.ExecCommand("ping", "127.0.0.1").Stdout() ``` -Note that `Exec` runs the command concurrently: it doesn't wait for the command to complete before returning any output. That's good, because this `ping` command will run forever (or until we get bored). +Note that `ExecCommand` runs the command concurrently: it doesn't wait for the command to complete before returning any output. That's good, because this `ping` command will run forever (or until we get bored). Instead, when we read from the pipe using `Stdout`, we see each line of output as it's produced: @@ -178,7 +176,7 @@ The `func` we supply to `Filter` takes just two parameters: a reader to read fro If our `func` returns some error, then, just as with the `Do` example, the pipe's error status is set, and subsequent stages become a no-op. -Filters run concurrently, so the pipeline can start producing output before the input has been fully read, as it did in the `ping` example. In fact, most built-in pipe methods, including `Exec`, are implemented *using* `Filter`. +Filters run concurrently, so the pipeline can start producing output before the input has been fully read, as it did in the `ping` example. In fact, most built-in pipe methods, including `ExecCommand`, are implemented *using* `Filter`. If we want to scan input line by line, we could do that with a `Filter` function that creates a `bufio.Scanner` on its input, but we don't need to: @@ -237,7 +235,7 @@ If you're already familiar with shell scripting and the Unix toolset, here is a | Unix / shell | `script` equivalent | | ------------------ | ------------------- | -| (any program name) | [`Exec`](https://pkg.go.dev/github.com/bitfield/script#Exec) | +| (any program name) | [`ExecCommand`](https://pkg.go.dev/github.com/bitfield/script#ExecCommand) | | `[ -f FILE ]` | [`IfExists`](https://pkg.go.dev/github.com/bitfield/script#IfExists) | | `>` | [`WriteFile`](https://pkg.go.dev/github.com/bitfield/script#Pipe.WriteFile) | | `>>` | [`AppendFile`](https://pkg.go.dev/github.com/bitfield/script#Pipe.AppendFile) | @@ -256,6 +254,7 @@ If you're already familiar with shell scripting and the Unix toolset, here is a | `jq` | [`JQ`](https://pkg.go.dev/github.com/bitfield/script#Pipe.JQ) | | `ls` | [`ListFiles`](https://pkg.go.dev/github.com/bitfield/script#ListFiles) | | `sed` | [`Replace`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Replace) / [`ReplaceRegexp`](https://pkg.go.dev/github.com/bitfield/script#Pipe.ReplaceRegexp) | +| `sh` | [`Shell`](https://pkg.go.dev/github.com/bitfield/script#Shell) | | `sha256sum` | [`Hash`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Hash) / [`HashSums`](https://pkg.go.dev/github.com/bitfield/script#Pipe.HashSums) | | `tail` | [`Last`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Last) | | `tee` | [`Tee`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Tee) | @@ -306,13 +305,14 @@ These are functions that create a pipe with a given contents: | [`Args`](https://pkg.go.dev/github.com/bitfield/script#Args) | command-line arguments | | [`Do`](https://pkg.go.dev/github.com/bitfield/script#Do) | HTTP response | | [`Echo`](https://pkg.go.dev/github.com/bitfield/script#Echo) | a string | -| [`Exec`](https://pkg.go.dev/github.com/bitfield/script#Exec) | command output | +| [`ExecCommand`](https://pkg.go.dev/github.com/bitfield/script#ExecCommand) | command output | | [`File`](https://pkg.go.dev/github.com/bitfield/script#File) | file contents | | [`FindFiles`](https://pkg.go.dev/github.com/bitfield/script#FindFiles) | recursive file listing | | [`Get`](https://pkg.go.dev/github.com/bitfield/script#Get) | HTTP response | | [`IfExists`](https://pkg.go.dev/github.com/bitfield/script#IfExists) | do something only if some file exists | | [`ListFiles`](https://pkg.go.dev/github.com/bitfield/script#ListFiles) | file listing (including wildcards) | | [`Post`](https://pkg.go.dev/github.com/bitfield/script#Post) | HTTP response | +| [`Shell`](https://pkg.go.dev/github.com/bitfield/script#Shell) | command output, run via the system shell | | [`Slice`](https://pkg.go.dev/github.com/bitfield/script#Slice) | slice elements, one per line | | [`Stdin`](https://pkg.go.dev/github.com/bitfield/script#Stdin) | standard input | @@ -344,7 +344,7 @@ Filters are methods on an existing pipe that also return a pipe, allowing you to | [`Do`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Do) | response to supplied HTTP request | | [`Echo`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Echo) | all input replaced by given string | | [`EncodeBase64`](https://pkg.go.dev/github.com/bitfield/script#Pipe.EncodeBase64) | input encoded to base64 | -| [`Exec`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Exec) | filtered through external command | +| [`ExecCommand`](https://pkg.go.dev/github.com/bitfield/script#Pipe.ExecCommand) | filtered through external command | | [`ExecForEach`](https://pkg.go.dev/github.com/bitfield/script#Pipe.ExecForEach) | execute given command template for each line of input | | [`Filter`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Filter) | user-supplied function filtering a reader to a writer | | [`FilterLine`](https://pkg.go.dev/github.com/bitfield/script#Pipe.FilterLine) | user-supplied function filtering each line to a string| @@ -363,6 +363,7 @@ Filters are methods on an existing pipe that also return a pipe, allowing you to | [`RejectRegexp`](https://pkg.go.dev/github.com/bitfield/script#Pipe.RejectRegexp) | lines not matching given regexp | | [`Replace`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Replace) | matching text replaced with given string | | [`ReplaceRegexp`](https://pkg.go.dev/github.com/bitfield/script#Pipe.ReplaceRegexp) | matching text replaced with given string | +| [`Shell`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Shell) | filtered through the system shell | | [`Tee`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Tee) | input copied to supplied writers | Note that filters run concurrently, rather than producing nothing until each stage has fully read its input. This is convenient for executing long-running commands, for example. If you do need to wait for the pipeline to complete, call [`Wait`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Wait). @@ -388,6 +389,8 @@ Sinks are methods that return some data from a pipe, ending the pipeline and ext | Version | New | | ----------- | ------- | +| 0.25.0 | [`ExecCommand`](https://pkg.go.dev/github.com/bitfield/script#Pipe.ExecCommand) / [`Shell`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Shell) supersede `Exec` (thanks [Dhanalakshmi-D04](https://github.com/Dhanalakshmi-D04)) | +| | [`WithContext`](https://pkg.go.dev/github.com/bitfield/script#Pipe.WithContext) (thanks [billvamva](https://github.com/billvamva)) | | 0.24.1 | [`JQ`](https://pkg.go.dev/github.com/bitfield/script#Pipe.JQ) accepts JSONLines data | | 0.24.0 | [`Hash`](https://pkg.go.dev/github.com/bitfield/script#Pipe.Hash) | | | [`HashSums`](https://pkg.go.dev/github.com/bitfield/script#Pipe.HashSums) | diff --git a/go.mod b/go.mod index 0e7c1c6..ec68829 100644 --- a/go.mod +++ b/go.mod @@ -2,7 +2,7 @@ module github.com/bitfield/script go 1.25.0 -toolchain go1.26.4 +toolchain go1.26.6 require ( github.com/google/go-cmp v0.5.9 diff --git a/script.go b/script.go index 457505d..1b32722 100644 --- a/script.go +++ b/script.go @@ -18,6 +18,7 @@ import ( "os/exec" "path/filepath" "regexp" + "runtime" "sort" "strconv" "strings" @@ -59,16 +60,24 @@ func Echo(s string) *Pipe { return NewPipe().WithReader(strings.NewReader(s)) } -// Exec creates a pipe that runs cmdLine as an external command and produces -// its combined output (interleaving standard output and standard error). See -// [Pipe.Exec] for error handling details. +// Exec creates a pipe that runs cmdLine as an external command. // -// Use [Pipe.Exec] to send the contents of an existing pipe to the command's -// standard input. +// Prefer using [Shell] or [ExecCommand] instead unless you specifically need +// [shell.Fields]-style parsing behaviour. +// +// See [Pipe.Exec] for details. func Exec(cmdLine string) *Pipe { return NewPipe().Exec(cmdLine) } +// ExecCommand creates a pipe that runs name as an external command with the supplied +// args. +// +// See [Pipe.ExecCommand] for details. +func ExecCommand(name string, args ...string) *Pipe { + return NewPipe().ExecCommand(name, args...) +} + // File creates a pipe that reads from the file path. func File(path string) *Pipe { f, err := os.Open(path) @@ -119,12 +128,13 @@ func Get(url string) *Pipe { return NewPipe().Get(url) } -// IfExists tests whether path exists, and creates a pipe whose error status -// reflects the result. If the file doesn't exist, the pipe's error status will -// be set, and if the file does exist, the pipe will have no error status. This -// can be used to do some operation only if a given file exists: +// IfExists tests whether path exists. +// +// If the file doesn't exist, the pipe's error status will be set, and if the file does +// exist, the pipe will have no error status. This can be used to do some operation only +// if a given file exists, by chaining [Pipe.ExecCommand] afterwards: // -// IfExists("/foo/bar").Exec("/usr/bin/something") +// IfExists("/foo/bar").ExecCommand("/usr/bin/something") func IfExists(path string) *Pipe { _, err := os.Stat(path) if err != nil { @@ -187,6 +197,13 @@ func Post(url string) *Pipe { return NewPipe().Post(url) } +// Shell creates a pipe that runs cmdLine as a command via the system shell. +// +// See [Pipe.Shell] for details. +func Shell(cmdLine string) *Pipe { + return NewPipe().Shell(cmdLine) +} + // Slice creates a pipe containing each element of s, one per line. If s is // empty or nil, then the pipe is empty. func Slice(s []string) *Pipe { @@ -270,7 +287,7 @@ func (p *Pipe) Column(col int) *Pipe { // // Or from the output of a command: // -// script.Exec("ls /var/app/config/").Concat().Stdout() +// script.ExecCommand("ls /var/app/config/").Concat().Stdout() // // Each input file will be closed once it has been fully read. If any of the // files can't be opened or read, Concat will simply skip these and carry on, @@ -417,41 +434,75 @@ func (p *Pipe) Error() error { return p.err } -// Exec runs cmdLine as an external command, sending it the contents of the -// pipe as input, and produces the command's standard output (see below for -// error output). The effect of this is to filter the contents of the pipe -// through the external command. +// Exec runs cmdLine as an external command. +// +// The command receives the contents of the pipe as its standard input, and the +// command's combined output (interleaving standard output and standard error) is sent +// to the pipe. +// +// # Argument parsing +// +// cmdLine is split into a program name and arguments using [shell.Fields]-style +// parsing. To pass the program name and arguments as separate parameters, use +// [Pipe.ExecCommand] instead. +// +// Because [shell.Fields] handles splitting, quoting, and variable expansion in a way +// that's slightly inconsistent with shells on various platforms, Exec will at some +// point be deprecated in favour of [Pipe.ExecCommand] (if you are constructing the +// command line from individual arguments) or [Pipe.Shell] (if the command line is a +// string). +// +// See [Pipe.ExecCommand] for details of error handling, context, and environment +// variables. +func (p *Pipe) Exec(cmdLine string) *Pipe { + args, err := shell.Fields(cmdLine, nil) + if err != nil { + return p.WithError(err) + } + return p.ExecCommand(args[0], args[1:]...) +} + +// ExecCommand runs name as an external command with the supplied args. +// +// The command receives the contents of the pipe as its standard input, and the +// command's combined output (interleaving standard output and standard error) is sent +// to the pipe. +// +// # Argument parsing +// +// The arguments are not parsed or expanded in any way; name and args are passed +// directly to the operating system, exactly as supplied, with no shell involved. This +// is the same API as Go's standard [os/exec.Command]. See also [Pipe.Shell] which +// passes the command line through the system shell for variable expansion, etc. // // # Environment // -// The command inherits the current process's environment, optionally modified -// by [Pipe.WithEnv]. +// The command inherits the current process's environment, or the pipe's environment if +// one was previously set using [Pipe.WithEnv]. // // # Context // -// The command inherits the pipe's context (if any was set by [Pipe.WithContext]), and -// will be cancelled if the context is cancelled or times out. +// The command inherits the pipe's context (if any was set by +// [Pipe.WithContext]), and will be cancelled if the context is +// cancelled or times out. // // # Error handling // -// If the command had a non-zero exit status, the pipe's error status will also -// be set to the string “exit status X”, where X is the integer exit status. -// Even in the event of a non-zero exit status, the command's output will still -// be available in the pipe. This is often helpful for debugging. However, -// because [Pipe.String] is a no-op if the pipe's error status is set, if you -// want output you will need to reset the error status before calling -// [Pipe.String]. -// -// If the command writes to its standard error stream, this will also go to the -// pipe, along with its standard output. However, the standard error text can -// instead be redirected to a supplied writer, using [Pipe.WithStderr]. -func (p *Pipe) Exec(cmdLine string) *Pipe { +// If the command had a non-zero exit status, the pipe's error status +// will also be set to the string "exit status X", where X is the +// integer exit status. Even in the event of a non-zero exit status, +// the command's output will still be available in the pipe. This is +// often helpful for debugging. However, because [Pipe.String] is a +// no-op if the pipe's error status is set, if you want output you +// will need to reset the error status before calling [Pipe.String]. +// +// If the command writes to its standard error stream, this will also +// go to the pipe, along with its standard output. However, the +// standard error text can instead be redirected to a supplied writer, +// using [Pipe.WithStderr]. +func (p *Pipe) ExecCommand(name string, args ...string) *Pipe { return p.Filter(func(r io.Reader, w io.Writer) error { - args, err := shell.Fields(cmdLine, nil) - if err != nil { - return err - } - cmd := exec.CommandContext(p.ctx, args[0], args[1:]...) + cmd := exec.CommandContext(p.ctx, name, args...) cmd.Stdin = r cmd.Stdout = w cmd.Stderr = w @@ -463,7 +514,7 @@ func (p *Pipe) Exec(cmdLine string) *Pipe { if pipeEnv != nil { cmd.Env = pipeEnv } - err = cmd.Start() + err := cmd.Start() if err != nil { fmt.Fprintln(cmd.Stderr, err) return err @@ -472,10 +523,11 @@ func (p *Pipe) Exec(cmdLine string) *Pipe { }) } -// ExecForEach renders cmdLine as a Go template for each line of input, running -// the resulting command, and produces the combined output of all these -// commands in sequence. See [Pipe.Exec] for details on error handling and -// environment variables. +// ExecForEach runs cmdLine for each line of input. +// +// cmdLine is rendered as a Go template for each line of input, running the resulting +// command, and produces the combined output of all these commands in sequence. See +// [Pipe.ExecCommand] for details on error handling and environment variables. // // This is mostly useful for substituting data into commands using Go template // syntax. For example: @@ -518,8 +570,9 @@ func (p *Pipe) ExecForEach(cmdLine string) *Pipe { if pipeStderr != nil { cmd.Stderr = pipeStderr } - if p.env != nil { - cmd.Env = p.env + pipeEnv := p.environment() + if pipeEnv != nil { + cmd.Env = pipeEnv } err = cmd.Start() if err != nil { @@ -538,9 +591,9 @@ func (p *Pipe) ExecForEach(cmdLine string) *Pipe { var exitStatusPattern = regexp.MustCompile(`exit status (\d+)$`) -// ExitStatus returns the integer exit status of a previous command (for -// example run by [Pipe.Exec]). This will be zero unless the pipe's error -// status is set and the error matches the pattern “exit status %d”. +// ExitStatus returns the integer exit status of a previous command (for example run by +// [Pipe.ExecCommand]). This will be zero unless the pipe's error status is set and the +// error matches the pattern “exit status %d”. func (p *Pipe) ExitStatus() int { if p.Error() == nil { return 0 @@ -931,6 +984,33 @@ func (p *Pipe) SHA256Sums() *Pipe { return p.HashSums(sha256.New()) } +// Shell runs cmdLine as a command via the operating system's standard shell. +// +// cmdLine will be passed to the shell process for expansion and execution ("sh -c" on +// Unix-like systems, "cmd /C" on Windows). The command will receive the contents of the +// pipe as input, and the command's combined output will be sent to the pipe. +// +// # Argument parsing +// +// cmdLine is passed to the shell completely unmodified as a single argument; Shell +// performs no parsing of cmdLine at all, so the shell alone is responsible for +// interpreting quoting, variable expansion, and other syntax, exactly as it would on an +// interactive command line. Shell is preferred over [Pipe.Exec] because it eliminates +// inconsistencies in quoting and expansion. +// +// Note that variable syntax differs by platform: Unix shells expand variables written +// as $VAR, while cmd.exe on Windows expands variables written as %VAR%. +// +// See [Pipe.ExecCommand] for details on error handling, context, and environment +// variables set via [Pipe.WithEnv]. +func (p *Pipe) Shell(cmdLine string) *Pipe { + shell, flag := "sh", "-c" + if runtime.GOOS == "windows" { + shell, flag = "cmd", "/C" + } + return p.ExecCommand(shell, flag, cmdLine) +} + // Slice returns the pipe's contents as a slice of strings, one element per // line, or an error. // @@ -945,9 +1025,11 @@ func (p *Pipe) Slice() ([]string, error) { return result, p.Error() } -// stdErr returns the pipe's configured standard error writer for commands run -// via [Pipe.Exec] and [Pipe.ExecForEach]. The default is nil, which means that -// error output will go to the pipe. +// stdErr returns the pipe's configured standard error writer. +// +// This where standard error output will go for commands run via [Pipe.Shell], +// [Pipe.ExecCommand], or [Pipe.ExecForEach]. The default is nil, which means that error +// output will go directly to the pipe. func (p *Pipe) stdErr() io.Writer { if p.mu == nil { // uninitialised pipe return nil @@ -1006,15 +1088,17 @@ func (p *Pipe) Wait() error { return p.Error() } -// WithContext sets the context for subsequent [Pipe.Exec], [Pipe.ExecForEach], [Pipe.Get] and [Pipe.Post] commands. +// WithContext sets the context inherited by subsequent [Pipe.ExecCommand], +// [Pipe.ExecForEach], [Pipe.Get], [Pipe.Post], and [Pipe.Shell] commands. func (p *Pipe) WithContext(ctx context.Context) *Pipe { p.ctx = ctx return p } -// WithEnv sets the environment for subsequent [Pipe.Exec] and [Pipe.ExecForEach] -// commands to the string slice env, using the same format as [os/exec.Cmd.Env]. -// An empty slice unsets all existing environment variables. +// WithEnv sets the environment inherited by subsequent [Pipe.ExecCommand] and +// [Pipe.ExecForEach], and [Pipe.Shell] commands to the string slice env, using the same +// format as [os/exec.Cmd.Env]. An empty slice unsets all existing environment +// variables. func (p *Pipe) WithEnv(env []string) *Pipe { p.mu.Lock() defer p.mu.Unlock() @@ -1047,8 +1131,10 @@ func (p *Pipe) WithReader(r io.Reader) *Pipe { return p } -// WithStderr sets the standard error output for [Pipe.Exec] or -// [Pipe.ExecForEach] commands to w, instead of the pipe. +// WithStderr sets the standard error output writer for commands. +// +// The standard error output of commands run by [Pipe.ExecCommand], [Pipe.ExecForEach], +// and [Pipe.Shell] commands will go to w, instead of the pipe as they would otherwise. func (p *Pipe) WithStderr(w io.Writer) *Pipe { p.mu.Lock() defer p.mu.Unlock() diff --git a/script_test.go b/script_test.go index c4f9a77..69a6dd8 100644 --- a/script_test.go +++ b/script_test.go @@ -353,10 +353,10 @@ func TestExecForEach_SendsStderrOutputToPipeStderr(t *testing.T) { } } -func TestExecSendsStderrOutputToPipeStderr(t *testing.T) { +func TestExecCommand_SendsStderrOutputToPipeStderr(t *testing.T) { t.Parallel() buf := new(bytes.Buffer) - out, err := script.NewPipe().WithStderr(buf).Exec("go").String() + out, err := script.NewPipe().WithStderr(buf).ExecCommand("go").String() if err == nil { t.Fatal("want error when command returns a non-zero exit status") } @@ -1190,6 +1190,54 @@ func TestSHA256Sums_OutputsCorrectHashForEachSpecifiedFile(t *testing.T) { } } +func TestShellErrorsRunningCommandThatDoesNotExist(t *testing.T) { + t.Parallel() + p := script.Shell("doesntexist_command_xyz") + p.Wait() + if p.Error() == nil { + t.Error("want error running non-existent command") + } +} + +func TestShellIsNoOpOnPipeWithExistingError(t *testing.T) { + t.Parallel() + fakeErr := errors.New("existing error") + p := script.NewPipe().WithError(fakeErr).Shell("echo hello") + if p.Error() != fakeErr { + t.Errorf("want existing error %v preserved, got %v", fakeErr, p.Error()) + } +} + +func TestShellRunsShWithEchoHelloAndGetsOutputHello(t *testing.T) { + t.Parallel() + p := script.Shell("echo hello") + if p.Error() != nil { + t.Fatal(p.Error()) + } + got, err := p.String() + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(got, "hello") { + t.Error(got) + } +} + +func TestShellSendsStderrOutputToPipeStderr(t *testing.T) { + t.Parallel() + buf := new(bytes.Buffer) + out, err := script.NewPipe().WithStderr(buf).Shell("go").String() + if err == nil { + t.Fatal("want error when command returns a non-zero exit status") + } + if out != "" { + t.Fatalf("unexpected output: %q", out) + } + if !strings.Contains(buf.String(), "Usage") { + t.Errorf("want stderr output containing the word 'Usage', got %q", buf.String()) + } +} + func TestTeeUsesConfiguredStdoutAsDefault(t *testing.T) { t.Parallel() buf := new(bytes.Buffer) @@ -1223,21 +1271,29 @@ func TestTeeWritesDataToSuppliedWritersAsWellAsToPipe(t *testing.T) { } } -func TestExecErrorsWhenTheSpecifiedCommandDoesNotExist(t *testing.T) { +func TestExecCommand_ErrorsWhenTheSpecifiedCommandDoesNotExist(t *testing.T) { t.Parallel() - p := script.Exec("doesntexist") + p := script.ExecCommand("doesntexist") p.Wait() if p.Error() == nil { t.Error("want error running non-existent command") } } -func TestExecRunsGoWithNoArgsAndGetsUsageMessagePlusErrorExitStatus2(t *testing.T) { +func TestExecSetsErrorImmediatelyOnInvalidCommandLineSyntax(t *testing.T) { + t.Parallel() + p := script.Exec("echo \"unterminated") + if p.Error() == nil { + t.Error("want error to be set immediately after Exec with invalid syntax, before reading the pipe") + } +} + +func TestExecCommand_RunsGoWithNoArgsAndGetsUsageMessagePlusErrorExitStatus2(t *testing.T) { t.Parallel() // We can't make many cross-platform assumptions about what external // commands will be available, but it seems logical that 'go' would be // (though it may not be in the user's path) - p := script.Exec("go") + p := script.ExecCommand("go") output, err := p.String() if err == nil { t.Fatal("want error when command returns a non-zero exit status") @@ -1252,9 +1308,9 @@ func TestExecRunsGoWithNoArgsAndGetsUsageMessagePlusErrorExitStatus2(t *testing. } } -func TestExecRunsGoHelpAndGetsUsageMessage(t *testing.T) { +func TestExecComnmand_RunsGoHelpAndGetsUsageMessage(t *testing.T) { t.Parallel() - p := script.Exec("go help") + p := script.ExecCommand("go", "help") if p.Error() != nil { t.Fatal(p.Error()) } @@ -1828,40 +1884,6 @@ func TestWithStdout_SetsSpecifiedWriterAsStdout(t *testing.T) { } } -func TestWithEnv_UnsetsAllEnvVarsGivenEmptySlice(t *testing.T) { - t.Parallel() - p := script.NewPipe().WithEnv([]string{"ENV1=test1"}).Exec("sh -c 'echo ENV1=$ENV1'") - want := "ENV1=test1\n" - got, err := p.String() - if err != nil { - t.Fatal(err) - } - if got != want { - t.Fatalf("want %q, got %q", want, got) - } - got, err = p.Echo("").WithEnv([]string{}).Exec("sh -c 'echo ENV1=$ENV1'").String() - if err != nil { - t.Fatal(err) - } - want = "ENV1=\n" - if got != want { - t.Errorf("want %q, got %q", want, got) - } -} - -func TestWithEnv_SetsGivenVariablesForSubsequentExec(t *testing.T) { - t.Parallel() - env := []string{"ENV1=test1", "ENV2=test2"} - got, err := script.NewPipe().WithEnv(env).Exec("sh -c 'echo ENV1=$ENV1 ENV2=$ENV2'").String() - if err != nil { - t.Fatal(err) - } - want := "ENV1=test1 ENV2=test2\n" - if got != want { - t.Errorf("want %q, got %q", want, got) - } -} - func TestErrorReturnsErrorSetByPreviousPipeStage(t *testing.T) { t.Parallel() p := script.File("testdata/nonexistent.txt") @@ -2065,9 +2087,9 @@ func TestEncodeBase64_CorrectlyEncodesInputBytes(t *testing.T) { } } -func TestWithStdErr_IsConcurrencySafeAfterExec(t *testing.T) { +func TestWithStdErr_IsConcurrencySafeAfterExecCommand(t *testing.T) { t.Parallel() - err := script.Exec("echo").WithStderr(nil).Wait() + err := script.ExecCommand("echo").WithStderr(nil).Wait() if err != nil { t.Fatal(err) } @@ -2178,11 +2200,11 @@ func TestHashSums_OutputsEmptyStringForFileThatCannotBeHashed(t *testing.T) { } } -func TestWithContext_SkipsExecWithCancelledContext(t *testing.T) { +func TestWithContext_SkipsExecCommandWithCancelledContext(t *testing.T) { t.Parallel() ctx, cancel := context.WithTimeout(context.Background(), time.Second) cancel() - output, err := script.WithContext(ctx).Exec("echo Hello, World").String() + output, err := script.WithContext(ctx).ExecCommand("echo", "Hello, World").String() if err == nil { t.Error("expected error due to context cancellation, got nil") } @@ -2253,16 +2275,16 @@ func ExampleEcho() { // Hello, world! } -func ExampleExec_exit_status_zero() { - p := script.Exec("echo") +func ExampleExecCommand_exit_status_zero() { + p := script.ExecCommand("echo") p.Wait() fmt.Println(p.ExitStatus()) // Output: // 0 } -func ExampleExec_exit_status_not_zero() { - p := script.Exec("false") +func ExampleExecCommand_exit_status_not_zero() { + p := script.ExecCommand("false") p.Wait() fmt.Println(p.ExitStatus()) // Output: @@ -2394,7 +2416,7 @@ func ExamplePipe_EncodeBase64() { } func ExamplePipe_ExitStatus() { - p := script.Exec("echo") + p := script.ExecCommand("echo") fmt.Println(p.ExitStatus()) // Output: // 0 @@ -2691,12 +2713,18 @@ func ExamplePipe_WithContext() { func ExamplePipe_WithStderr() { buf := new(bytes.Buffer) - script.NewPipe().WithStderr(buf).Exec("go").Wait() + script.NewPipe().WithStderr(buf).ExecCommand("go").Wait() fmt.Println(strings.Contains(buf.String(), "Usage")) // Output: // true } +func ExampleShell() { + script.Shell("echo Hello, world!").Stdout() + // Output: + // Hello, world! +} + func ExampleSlice() { input := []string{"1", "2", "3"} script.Slice(input).Stdout() diff --git a/script_unix_test.go b/script_unix_test.go index ae512c4..cd93ae0 100644 --- a/script_unix_test.go +++ b/script_unix_test.go @@ -7,6 +7,8 @@ import ( "path/filepath" "testing" + "strings" + "github.com/bitfield/script" "github.com/google/go-cmp/cmp" ) @@ -22,47 +24,6 @@ func TestExecForEach_HandlesLongLines(t *testing.T) { } } -func TestExecRunsShWithEchoHelloAndGetsOutputHello(t *testing.T) { - t.Parallel() - p := script.Exec("sh -c 'echo hello'") - if p.Error() != nil { - t.Fatal(p.Error()) - } - want := "hello\n" - got, err := p.String() - if err != nil { - t.Fatal(err) - } - if want != got { - t.Error(cmp.Diff(want, got)) - } -} - -func TestExecRunsShWithinShWithEchoInceptionAndGetsOutputInception(t *testing.T) { - t.Parallel() - p := script.Exec("sh -c 'sh -c \"echo inception\"'") - if p.Error() != nil { - t.Fatal(p.Error()) - } - want := "inception\n" - got, err := p.String() - if err != nil { - t.Fatal(err) - } - if want != got { - t.Error(cmp.Diff(want, got)) - } -} - -func TestExecErrorsRunningShellCommandWithUnterminatedStringArgument(t *testing.T) { - t.Parallel() - p := script.Exec("sh -c 'echo oh no") - p.Wait() - if p.Error() == nil { - t.Error("want error running 'sh' command line containing unterminated string") - } -} - func TestExecForEach_RunsEchoWithABCAndGetsOutputABC(t *testing.T) { t.Parallel() p := script.Echo("a\nb\nc\n").ExecForEach("echo {{.}}") @@ -95,9 +56,9 @@ func TestExecForEach_CorrectlyEvaluatesTemplateContainingIfStatement(t *testing. } } -func TestExecPipesDataToExternalCommandAndGetsExpectedOutput(t *testing.T) { +func TestExecCommandPipesDataToExternalCommandAndGetsExpectedOutput(t *testing.T) { t.Parallel() - p := script.File("testdata/hello.txt").Exec("cat") + p := script.File("testdata/hello.txt").ExecCommand("cat") want := "hello world" got, err := p.String() if err != nil { @@ -129,8 +90,8 @@ func TestFindFiles_DoesNotErrorWhenSubDirectoryIsNotReadable(t *testing.T) { } } -func ExampleExec_ok() { - script.Exec("echo Hello, world!").Stdout() +func ExampleExecCommand_ok() { + script.ExecCommand("echo", "Hello, world!").Stdout() // Output: // Hello, world! } @@ -147,13 +108,13 @@ func ExampleFindFiles() { } func ExampleIfExists_exec() { - script.IfExists("./testdata/hello.txt").Exec("echo hello").Stdout() + script.IfExists("./testdata/hello.txt").ExecCommand("echo", "hello").Stdout() // Output: // hello } func ExampleIfExists_noExec() { - script.IfExists("doesntexist").Exec("echo hello").Stdout() + script.IfExists("doesntexist").ExecCommand("echo", "hello").Stdout() // Output: // } @@ -209,8 +170,8 @@ func ExamplePipe_Dirname() { // C: } -func ExamplePipe_Exec() { - script.Echo("Hello, world!").Exec("tr a-z A-Z").Stdout() +func ExamplePipe_ExecCommand() { + script.Echo("Hello, world!").ExecCommand("tr", "a-z", "A-Z").Stdout() // Output: // HELLO, WORLD! } @@ -222,3 +183,95 @@ func ExamplePipe_ExecForEach() { // b // c } + +func ExamplePipe_Shell() { + script.Echo("Hello, world!").Shell("tr a-z A-Z").Stdout() + // Output: + // HELLO, WORLD! +} + +func TestShell_ExpandsEnvironmentVariablesSetViaWithEnv(t *testing.T) { + t.Parallel() + env := []string{"ENV1=test1", "ENV2=test2"} + got, err := script.NewPipe().WithEnv(env).Shell("echo ENV1=$ENV1 ENV2=$ENV2").String() + if err != nil { + t.Fatal(err) + } + want := "ENV1=test1 ENV2=test2\n" + if want != got { + t.Error(cmp.Diff(want, got)) + } +} + +func TestShellExpandsHomeVariableWithoutWithEnv(t *testing.T) { + t.Parallel() + p := script.Shell("echo $HOME") + if p.Error() != nil { + t.Fatal(p.Error()) + } + got, err := p.String() + if err != nil { + t.Fatal(err) + } + if strings.TrimSpace(got) == "" { + t.Error("want non-empty $HOME expansion, got empty string") + } +} + +func TestShellPipesDataToExternalCommandAndGetsExpectedOutput(t *testing.T) { + t.Parallel() + p := script.File("testdata/hello.txt").Shell("cat") + want := "hello world" + got, err := p.String() + if err != nil { + t.Fatal(err) + } + if want != got { + t.Error(cmp.Diff(want, got)) + } +} + +func TestShellOnEmptyPipeProducesNoOutputAndNoError(t *testing.T) { + t.Parallel() + got, err := script.NewPipe().Shell("cat").String() + if err != nil { + t.Fatal(err) + } + if got != "" { + t.Errorf("want empty output, got %q", got) + } +} + +func TestWithEnv_SetsGivenVariablesForSubsequentExec(t *testing.T) { + t.Parallel() + env := []string{"ENV1=test1", "ENV2=test2"} + got, err := script.NewPipe().WithEnv(env).Shell("echo ENV1=$ENV1 ENV2=$ENV2").String() + if err != nil { + t.Fatal(err) + } + want := "ENV1=test1 ENV2=test2\n" + if got != want { + t.Errorf("want %q, got %q", want, got) + } +} + +func TestWithEnv_UnsetsAllEnvVarsGivenEmptySlice(t *testing.T) { + t.Parallel() + p := script.NewPipe().WithEnv([]string{"ENV1=test1"}).Shell("echo ENV1=$ENV1") + want := "ENV1=test1\n" + got, err := p.String() + if err != nil { + t.Fatal(err) + } + if got != want { + t.Fatalf("want %q, got %q", want, got) + } + got, err = p.Echo("").WithEnv([]string{}).Shell("echo ENV1=$ENV1").String() + if err != nil { + t.Fatal(err) + } + want = "ENV1=\n" + if got != want { + t.Errorf("want %q, got %q", want, got) + } +} diff --git a/script_windows_test.go b/script_windows_test.go index 3290d4e..6fb4470 100644 --- a/script_windows_test.go +++ b/script_windows_test.go @@ -91,3 +91,16 @@ func ExamplePipe_Dirname() { // ./src // C:\ } + +func TestShell_ExpandsEnvironmentVariablesSetViaWithEnvOnWindows(t *testing.T) { + t.Parallel() + env := []string{"ENV1=test1"} + got, err := script.NewPipe().WithEnv(env).Shell("echo ENV1=%ENV1%").String() + if err != nil { + t.Fatal(err) + } + want := "ENV1=test1\r\n" + if want != got { + t.Errorf("want %q, got %q", want, got) + } +}