Summary: Doing this before we actually remove the build field is kinder to users.
Test Plan:
This dumps a bunch of junk into our test output, because we still use
the `build` field. I don't think it's worth putting in a bunch of extra
logic to catch that. This won't be around for very long, and this will
be an incentive to clean it up.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D165
Summary:
Replace use of `build` with the new `unpack` field, and clean up the
type section.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D155
Summary: Package for 0.1.2ubuntu~utopic1-1 is live on Launchpad. Bumping changelog.
Reviewers: jacko
Reviewed By: jacko
Differential Revision: https://phabricator.buildinspace.com/D164
Summary:
Provides native support for tar and zip unpacking. Python documentation
on `tarfile` warns about using the `extractall` method on untrusted
archives, so we try to check for the evil paths that it mentions.
Test Plan: New tests.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D154
Summary:
Doing tar and zip unpacking looks like it's going to need temporary
files. We could expect plugins to use the global temp dir, but so far
we've been trying to avoid that. @sean let me know what you think.
Reviewers: sean
Reviewed By: sean
Subscribers: sean
Differential Revision: https://phabricator.buildinspace.com/D153
Summary:
The number of context managers in that function is getting annoying,
because of all the nested with-statements. And we're about to add
another in the next diff for tmp dirs. Contextlib provides an ExitStack
class to help deal with lots of context managers.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D152
Summary:
Add a "source version" when generating source packages for Ubuntu. The source
version is a part of the software version, which is encoded in the original
tarball. This allows some flexibility when uploading to Launchpad, which will
reject changes for the same software version. For example, the source version
can be bumped to upload a fix for a broken package (without bumping the actual
software version). Note that the package version cannot be used to accomplish
this; it is not part of the source tarball, so bumping this version does not
pervent Launchpad from rejecting the changes.
Rename the script, since this is a bit specific for Ubuntu and Launchpad.
See: http://askubuntu.com/a/391751
Reviewers: jacko
Reviewed By: jacko
Differential Revision: https://phabricator.buildinspace.com/D163
Summary:
Important changes in this release:
- fix curl plugin's quote handling (D144)
- support user-defined plugins (D141)
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D147
Summary:
Our Arch package is broken because it doesn't install asyncio and
pathlib. These shouldn't *need* to be installed, because Arch is running
Python 3.4, but the executable script that setup.py generates is getting
confused. It seems like the right thing to do is to make sure these
extra dependencies never make it into our `install_requires` list in
Python 3.4. I *think* the fact that our setup.py ends up in the source
distribution means that `pip install peru` will still do the right thing
for Python 3.3.
Test Plan: Tested `pip install .` in a python3.3 virtualenv.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D151
Summary:
Get rid of the separate install instructions for python3.3, and add
links at the bottom.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D149
Summary:
Add Debian packaging metadata and a script to generate a basic source package
that can be uploaded to a Launchpad PPA.
Reviewers: jacko
Reviewed By: jacko
Differential Revision: https://phabricator.buildinspace.com/D133
Summary:
Remove the third-party/ directory and no longer rely on copies of dependencies
in the repo. When needed, fetch dependencies normally. This can usually be
accomplished via `pip install -r requirements.txt` or `pip install -r
requirements-dev.txt`.
Even though `peru.yaml` is gone, `.peru/` is still in `.gitignore`, because the
tests check for untracked files and will detect `.peru/` as an error.
Test Plan: Successfully ran tests locally and via Travis CI.
Reviewers: jacko
Reviewed By: jacko
Differential Revision: https://phabricator.buildinspace.com/D145
Summary:
Instead, only create the dir when we need to actually write a
`lastimports` file. This is a short term fix for our parser tests
inadvertently creating a `.peru` directory in our tests dir. With this
fix, D145 should be able to remove `.peru` from our `.gitignore` file.
In the long term, it's clear that we have an organization problem. The
parser should obviously not be calling code that writes to disk. I think
the best solution will be to tear up the LocalModule class. It's a
holdover from when we thought we would be doing much more complicated
things to override dirs, like applying imports and running build
commands. Going forward, override dirs are going to be a much simpler
concept, and we can probably separate the parser from the code that
deals with the local project dir in the process of simplifying it.
Tracking issue: https://github.com/buildinspace/peru/issues/72
Test Plan:
`arc patch D145`, remove `.peru` from `.gitignore`, clean the repo, and
run tests. Tests pass without any warning about dirty files.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D146
Summary:
We have peru.sh and peru.bat, and also test.sh and test.bat. Rather than
maintaining both of those duplicates to support Windows, just
reimplement them in Python.
There's also validate-third-party.sh, but we never really use that on
Windows. We can port it in a later diff if we feel like it.
Test Plan:
Check that tests actually fail after introducing a bug. (To make sure
we're not actually testing the installed version instead of the repo
version.) Check that the linter fails after dirtying up some code. Check
that the untracked file test fails by creating a random file during
tests.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D143
Summary:
Add search directories for plugins. On Linux/Mac we use:
1. `$XDG_CONFIG_HOME/peru/plugins`
2. `/usr/local/lib/peru/plugins`
3. `/usr/lib/peru/plugins`
And on Windows it's:
1. `%LOCALAPPDATA%\peru\plugins`
2. `%PROGRAMFILES%\peru\plugins`
Get rid of the old rule that says you can't override existing plugins --
that was from back when we allowed individual projects to provide their
own plugins, which we no longer do.
Test Plan:
We had some old testing logic for custom plugin paths, even though that
feature had been mostly removed. Repurpose that test to load plugins
from a fake home directory. Manually test on Windows.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D141
Summary: Looking at them now, some of the help summaries are confusing.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D142
Summary:
A first shot at documenting how peru works internally, particularly the
plugin interface.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D139
Summary: We forgot to point .travis.yml to the new test script name.
Test Plan: Pushed a branch to my fork.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D140
Summary:
This is a cosmetic change and moves some more "targeted" scripts to specific
directories. It also pairs Windows-specific scripts with their Posix
counterparts.
Reviewers: jacko
Reviewed By: jacko
Differential Revision: https://phabricator.buildinspace.com/D138
Summary:
This was the recommendation of
https://packaging.python.org/en/latest/distributing.html, but I unwisely
ignored it back when I wrote our setup.py. It turns out that getting a
launcher script to work properly on Windows is tricky. The
`entry_points` field generates a platform-appropriate launcher script at
install time, so we get Windows for free.
Previously our launcher script checked the version of Python and printed
a helpful error if it was too low. Because our main.py has `yield from`
syntax in it, printing such an error message isn't possible; we'll crash
with a syntax error first. We could create another file within the
modile (or maybe an __init__.py) to support this, but for now I'm just
dropping the feature.
Test Plan:
Manually played with peru.sh to make sure it still works. Ran
`pip install .` to check that the generated script works. Also tested
that on my Windows VM, and it works great. (It generates an exe file, in
fact.)
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D135
Summary:
Git doesn't expect backslashes in tree merge prefixes, but on Windows we
were incorrectly using those. That got things very confused when
anything had an import path with slashes in it. Use pathlib to make sure
we always get a posix path for this.
Fixes https://github.com/buildinspace/peru/issues/60.
Test Plan: New unit test that was previously failing on Windows.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D136
Summary: Recommend pip, and include mentions of our AUR package and our PPA.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D134
Summary:
At some point we should do proper terminal capability detection, but
until then just disable the fancy stuff on Windows. I took a quick look
at Colorama, but that didn't seem to support the control codes we need.
Test Plan:
Ran a sync on Windows. Confirmed the default output is quiet and that
errors don't have junk around them.
Reviewers: sean
Reviewed By: sean
Differential Revision: https://phabricator.buildinspace.com/D131
Summary:
That way if a test has weird output, it's easier to tell what it's
coming from. Also allow test.sh to take command line arguments and
forward them to the unittest command.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D129
Summary:
Travis likes to pretend that its tests are running in a terminal, but it
doesn't support the ANSI control codes that our fancy display uses, so
we get a spew of crap in the build logs when peru runs inside of
validate_third_party.sh. Use `-q` to avoid the problem.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D127
Summary:
The script was originally written at `./scripts/packaging/arch/`, but we
moved the packaging directory to the toplevel before we committed it. I
forgot to change a path in the script, so it broke.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D128
Summary:
This version does a better job of detecting unsafe operations across
loops/threads in `PYTHONASYNCIODEBUG` mode, which is nice.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D123
Summary:
The rsync plugin is written in bash, so it won't work properly on Windows.
A couple field validation tests also depended on rsync, but those can use
the git plugin instead.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D122
Summary:
Eventually we might want some mechanism to allow multiple versions of
build commands, so that a rule can work on both Windows and POSIX. For
now, we'll just skip the tests that rely on build commands when we're
running on Windows.
https://github.com/buildinspace/peru/issues/17
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D121
Summary:
It looks like Windows takes an exclusive write lock on
`TemporaryNamedFile`s when they're created. We want to be able to open
temp files in a context manager than cleans them up automatically, which
is causing errors when the file is opened again for actual writing. It's
simpler to just use the temporary directory context manager and create
an unmanaged file in it.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D120
Summary:
The rules for building file URIs on Windows are a little more
complicated than just sticking `file://` in front of a path. Luckily
pathlib has a method that does everything right.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D118
Summary:
Lots of our test code dealing with repos was running commands as shell
strings for convenience. Even though we stuck to simple commands, we
still ran into issues with single-quoted parameters not working on
Windows. Skipping the shell entirely and using plain old exec solves
all these problems.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D117
Summary:
Before this change, tons of our tests failed on Windows because they
hardcoded paths of the form "a/b/c". Then `os.walk` spat out "a\b\c" and
everything would die. Pathlib solves this nicely.
I was tempted to make the whole shared test library (including e.g.
`create_dir`) return nothing but pathlib paths. Unfortunately, that
required callers to re-stringify paths before passing them to things
like subprocess `cwd` and `env` parameters. Getting rid of
`os.path.join` was nice, but it was more trouble than it was worth.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D115
Summary:
Because all asyncio subprocess output is read in binary mode, we don't
get universal newlines for free. But it's the right thing to do, because
we do all our printing with strings in text mode, which translates "\n"
back into the platform-appropriate line separator. So for example,
"\r\n" in a string on Windows will become "\r\r\n" when it gets printed.
This function ensures that all newlines are represented as "\n"
internally, which solves that problem and also helps our tests work on
Windows. Right now we only handle Windows, but we can expand this if
there's ever another newline style we have to support.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D114
Summary:
Launcher scripts for Windows. These probably shouldn't live in the top
level, but I haven't done the work to make them run from other
directories yet. I should do that before this lands.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D113
Summary:
This lets us hit Ctrl-C and still get results from tests that have run
so far.
Reviewers: sean
Differential Revision: https://phabricator.buildinspace.com/D112
Jack O'Connor