Opened 18 months ago
Closed 18 months ago
#29622 closed defect (fixed)
cygwin, conda, homebrew: Fixes for CI
Reported by:  mkoeppe  Owned by:  

Priority:  blocker  Milestone:  sage9.1 
Component:  porting: Cygwin  Keywords:  
Cc:  dimpase, slelievre, embray  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  Dima Pasechnik 
Report Upstream:  N/A  Work issues:  
Branch:  630576b (Commits, GitHub, GitLab)  Commit:  630576b0ced872081c792a3b339b3b505d65b404 
Dependencies:  #29530  Stopgaps: 
Description
Hopefully the last change for the CI for 9.1
Change History (26)
comment:1 Changed 18 months ago by
 Summary changed from cygwin: Fixes for CI to cygwin, conda: Fixes for CI
comment:2 Changed 18 months ago by
 Branch set to u/mkoeppe/cygwin__fixes_for_ci
comment:3 Changed 18 months ago by
 Commit set to f9700fd16d6faa0701088fd8f9743e0ee2d08207
 Dependencies set to #29530
 Summary changed from cygwin, conda: Fixes for CI to cygwin, conda, homebrew: Fixes for CI
comment:4 Changed 18 months ago by
This fixes some syntax errors in the cygwin build. Now it times out in stage 3. This would need to be rebalanced or split into several parallel stages
comment:5 Changed 18 months ago by
 Status changed from new to needs_review
comment:6 Changed 18 months ago by
 Commit changed from f9700fd16d6faa0701088fd8f9743e0ee2d08207 to 8df6b206ee3bd5a7b4bba676a8b6ef0c036bbe89
comment:7 Changed 18 months ago by
getting closer  https://github.com/mkoeppe/sage/runs/639064085
comment:8 Changed 18 months ago by
Fatal error: can't write 24 bytes to section .text of build/temp.cygwin3.1.4x86_643.7/build/cythonized/sage/rings/laurent_series_ring_element.o: 'No space left on device'
comment:9 Changed 18 months ago by
 Commit changed from 8df6b206ee3bd5a7b4bba676a8b6ef0c036bbe89 to ea92c284b5204560d65478e84c60806cc3a8c2ea
Branch pushed to git repo; I updated commit sha1. New commits:
ea92c28  Factor out .github/workflows/extrasagelocal.sh from cicygwin*.yml, remove archives after extracting

comment:10 Changed 18 months ago by
 Commit changed from ea92c284b5204560d65478e84c60806cc3a8c2ea to 0d4655355f32bb312dcda24292c9a5676428ad2b
Branch pushed to git repo; I updated commit sha1. New commits:
0d46553  Fixup

comment:11 Changed 18 months ago by
 Commit changed from 0d4655355f32bb312dcda24292c9a5676428ad2b to 9b1214fb51a1d9c5d5790d834ce406bbf51e6490
comment:12 Changed 18 months ago by
 Commit changed from 9b1214fb51a1d9c5d5790d834ce406bbf51e6490 to d3b5f8cb275fc7fd9b5aed8a86003c527d140ac2
comment:13 Changed 18 months ago by
This now manages to build Sage on Cygwin (see cygwinstageiii
in https://github.com/mkoeppe/sage/runs/640373882)
comment:14 Changed 18 months ago by
 Commit changed from d3b5f8cb275fc7fd9b5aed8a86003c527d140ac2 to 630576b0ced872081c792a3b339b3b505d65b404
Branch pushed to git repo; I updated commit sha1. New commits:
630576b  Update cygwinminimal, cygwinstandardpython2

comment:15 Changed 18 months ago by
Ready for review
comment:16 Changed 18 months ago by
 Cc slelievre embray added
cygwinminimal
, stageiia and stageiic still time out (https://github.com/mkoeppe/sage/runs/641053384); more rebalancing should be done in the next release cycle. I stop for this ticket and 9.1.
Uploading and downloading of our large sagelocal artifacts is somewhat brittle, a game of chance: For example, in https://github.com/mkoeppe/sage/runs/641209977 (cygwinstandard
), stageiii :
Exponential backoff for retry #5. Waiting for 33036.92082171844 milliseconds before continuing the upload at offset 1379926016 Total file count: 1  Processed file #0 (0.0%) Finished backoff for retry #5, continuing with upload Total file count: 1  Processed file #0 (0.0%) An error has been caught httpclient index 0, retrying the upload Error: read ECONNRESET at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) { errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' } Retry limit has been reached for chunk at offset 1379926016 to https://pipelines.actions.githubusercontent.com/4D1poTSDSSApU42GXYeGgQX6WjHwGPK9ZzdaE2SeHdlUs0ZBhW/_apis/resources/Containers/3201477?itemPath=sagelocalcommitb2c012bfed0fb9414e0db531acc43a3abf1ae1bacygwinstandard%5Csagelocaliii.tar ##[warning]Aborting upload for C:\tools\cygwin\tmp\sagelocaliii.tar due to failure Total size of all the files uploaded is 1379926016 bytes
In the same run, for https://github.com/mkoeppe/sage/runs/641394549 (cygwinstandardpython2
), the upload of stageiii succeeded,
and stageiva ... stageivd downloaded it successfully.
 stageiva (
make ptest
, https://github.com/mkoeppe/sage/runs/641394549) terminates, with a number of doctest errors and timeouts:sage t src/doc/en/constructions/plotting.rst # Timed out sage t src/doc/fr/tutorial/interfaces.rst # 1 doctest failed sage t src/doc/ja/tutorial/interfaces.rst # 2 doctests failed sage t src/doc/pt/tutorial/interfaces.rst # Timed out sage t src/doc/ru/tutorial/interfaces.rst # 6 doctests failed sage t src/sage/coding/binary_code.pyx # Timed out sage t src/sage/doctest/control.py # 1 doctest failed sage t src/sage/functions/log.py # 1 doctest failed sage t src/sage/functions/orthogonal_polys.py # 1 doctest failed sage t src/sage/functions/other.py # 2 doctests failed sage t src/sage/dynamics/arithmetic_dynamics/projective_ds.py # Timed out sage t src/sage/interfaces/gap.py # 1 doctest failed sage t src/sage/groups/class_function.py # Timed out sage t src/sage/interfaces/giac.py # 41 doctests failed sage t src/sage/libs/gap/element.pyx # 1 doctest failed sage t src/sage/libs/glpk/error.pyx # 2 doctests failed sage t src/sage/matrix/matrix1.pyx # 1 doctest failed sage t src/sage/numerical/backends/glpk_backend.pyx # 2 doctests failed sage t src/sage/parallel/map_reduce.py # Timed out after testing finished sage t src/sage/repl/interpreter.py # 3 doctests failed sage t src/sage/rings/number_field/number_field_element.pyx # Timed out sage t src/sage/sandpiles/sandpile.py # Timed out sage t src/sage/symbolic/constants.py # 3 doctests failed sage t src/sage/symbolic/expression_conversions.py # 1 doctest failed sage t src/sage/symbolic/expression.pyx # 1 doctest failed sage t src/sage/tests/benchmark.py # 2 doctests failed sage t src/sage/tests/books/computationalmathematicswithsagemath/sol/graphique_doctest.py # 1 doctest failed sage t src/sage/tests/cmdline.py # 3 doctests failed
Looks like the giac interface is broken.
 stageivb builds a few optional packages; I have noted build failures in some separate tickets.
 stageivc builds
cbc
,sage_numerical_backends_coin
successfully.
 stageivd (https://github.com/mkoeppe/sage/runs/641394565) builds a long list of optional packages, times out; I have noted build failures in some separate tickets.
comment:17 Changed 18 months ago by
Needs review. Let's get this into 9.1 please
comment:18 Changed 18 months ago by
 Priority changed from major to critical
comment:19 followup: ↓ 20 Changed 18 months ago by
 Reviewers set to Dima Pasechnik
 Status changed from needs_review to positive_review
this looks good; one minor question is how to get logs for particular failed spkg builds. Is the latter possible/documented, and I'm just not looking at the right place?
comment:20 in reply to: ↑ 19 Changed 18 months ago by
Replying to dimpase:
one minor question is how to get logs for particular failed spkg builds. Is the latter possible/documented, and I'm just not looking at the right place?
Do you see the artifact named "logs...." in https://github.com/mkoeppe/sage/actions/runs/94565308?
comment:21 followup: ↓ 23 Changed 18 months ago by
It contains both the logs of all packages, and the contents of $SAGE_LOCAL/var/tmp/sage/build/ (all failed packages)
comment:22 Changed 18 months ago by
Alternatively, if one does not want to download these huge log archives, one can open the section "Print out logs for immediate inspection" (for example in https://github.com/mkoeppe/sage/runs/641394520?check_suite_focus=true) to see logs of packages with errors. (This already works before artifacts are made available.)
comment:23 in reply to: ↑ 21 Changed 18 months ago by
Replying to mkoeppe:
It contains both the logs of all packages, and the contents of $SAGE_LOCAL/var/tmp/sage/build/ (all failed packages)
ok  I only looked at "view raw logs"  and these are, despite the name, less complete than artefacts, right?
comment:24 Changed 18 months ago by
"View raw logs" just gives you what is displayed in the right pane. This is a GitHub Actions feature. The log artefacts have a lot more, from my scripts.
comment:25 Changed 18 months ago by
 Priority changed from critical to blocker
comment:26 Changed 18 months ago by
 Branch changed from u/mkoeppe/cygwin__fixes_for_ci to 630576b0ced872081c792a3b339b3b505d65b404
 Resolution set to fixed
 Status changed from positive_review to closed
Last 10 new commits:
.github/workflows/tox*.yml: Make copying logs from containers more robust
tox.ini: Commit a failed build container so that logs are copied from it
toxoptional.yml: Fix likewise
cicygwin.yml: Split the file so that unrelated stages do not wait
Improve docker login instructions
Merge tag '9.1.rc2' into t/29530/make_docker_images_generated_by_github_ci_workflows_available
Rename cygwin workflows
cicygwin*.yml: Fix typos
tox.yml: Add dockercondaforge
tox.yml: Add homebrewmacospython3_xcodegcc_spkg