portman/newman not returning nonzero exit code when using "abortOnFailure"=false

[chyde80]

I am running in a Jenkins pipeline and am seeing "collection run completed" when the run actually has failed tests. Why is the exit code not coming back correctly w/ an error? I have "abortOnFailure" set to false so that I can run the entire collection w/ tests before failing the pipeline stage. I can see in the console that newman/portman is reporting "Collection run completed" instead of what I would expect from Newman's docs which should throw "collection run encountered an error", which I expect would throw a non-zero exit code.

[chyde80]

@thim81 @nicklloyd ?

[chyde80]

I had to write a custom workaround for this - I published results as junit, published them in jenkins using the junit plugin, and then parsed the # of failed tests. If >0, I set the build to failed. It would be cleaner if portman could correctly pass back a nonzero exit code when there are failed tests even when the run completes. Any updates on this?

[thim81]

@chyde80 Can you try this PR, that should provide a better exit code?
#530

[thim81]

@chyde80 did you had the chance to test the PR?

[chyde80]

I don't have a convenient way of testing until it's merged into the product - please do so if you're comfortable and I'll let you know. Or, publish it to an alternate version on NPM and I'll test on that version instead of latest.

[thim81]

hi @chyde80

it took a while before it landed but we just release Portman 1.33, which contains the improvement of the proper failure handling and exits of Portman.

We appreciate you taking the time to share this feedbac, it helps us make Portman better 🙌 .

If the new version, solves your issue, feel free to close this item.