Bugzilla – Full Text Bug Listing
| Summary: | [multi-process][breakpoints] Setting breakpoints on a running target does not work with multi-process | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Tools] CDT | Reporter: | Marc Khouzam <marc.khouzam> | ||||||||||||||||||||
| Component: | cdt-debug-dsf-gdb | Assignee: | Marc Khouzam <marc.khouzam> | ||||||||||||||||||||
| Status: | RESOLVED FIXED | QA Contact: | Marc Khouzam <marc.khouzam> | ||||||||||||||||||||
| Severity: | normal | ||||||||||||||||||||||
| Priority: | P3 | CC: | cdtdoug, cx186000, pawel.1.piech | ||||||||||||||||||||
| Version: | 8.0 | ||||||||||||||||||||||
| Target Milestone: | 8.0 | ||||||||||||||||||||||
| Hardware: | PC | ||||||||||||||||||||||
| OS: | Linux | ||||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||||
|
Description
Marc Khouzam
This issue can happen outside of multiple process when removing all breakpoints. In this case, multiple overlapping calls to IMIRunControl#executeWithTargetAvailable() are made also. See Bug 336899 *** Bug 336899 has been marked as a duplicate of this bug. *** I can also trigger this bug when grouping breakpoints in the debug view and then enabling an entire group while the program is running. For single process, we know that to be able to set a breakpoint, GDB requires one thread to be stopped. However, for multi-process, GDB requires a stopped thread of the process on which we want to set the breakpoint. This means that if we have to interrupt a thread to set a breakpoint, we need to pick a thread of the right process. (In reply to comment #4) > For single process, we know that to be able to set a breakpoint, GDB requires > one thread to be stopped. However, for multi-process, GDB requires a stopped > thread of the process on which we want to set the breakpoint. This means that > if we have to interrupt a thread to set a breakpoint, we need to pick a thread > of the right process. This comment applies to non-stop only, since in all-stop stopping any thread, will stop all other threads, of every process. Created attachment 192028 [details]
Tests for all-stop mode
The solution to this bug turned out to be filled with races, so I wrote 12 new tests to exercise the logic thoroughly.
Committed to HEAD.
Created attachment 192029 [details]
Fix for all-stop mode
This fix is for all-stop only. I still need to do it for non-stop mode.
This fix turned out to be pretty complex. Not only did we have to deal with concurrent calls to IMIRunControl.executeWithTargetAvailable, but I realized we also had nested calls to that method.
For example, when creating a watchpoint with a condition, we call executeWithTargetAvailable() to create the watchpoint and within that call, we again call executeWithTargetAvailable() to set the condition.
This had to be handled properly so that the second operation could complete to allow the first operation to complete. I added this scenario to the tests, to make sure it keeps on being supported.
There are some new APIs in this change to allow extenders of DSF-GDB to use it. Those changes are not necessary for the fix, but I think they will eventually be requested, so I added them now. I will email the list about changing the API (in a backwards compatible way) after API freeze.
I will commit this to HEAD.
*** cdt cvs genie on behalf of mkhouzam *** Bug 337893 - Tests for IMIRunControl.executeWithTargetAvailable, all-stop mode [*] Suite_Remote_7_0.java 1.5 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_0/Suite_Remote_7_0.java?root=Tools_Project&r1=1.4&r2=1.5 [+] MIRunControlTargetAvailableTest_7_0.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_0/MIRunControlTargetAvailableTest_7_0.java?root=Tools_Project&revision=1.1&view=markup [*] Suite_7_0.java 1.5 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_0/Suite_7_0.java?root=Tools_Project&r1=1.4&r2=1.5 [+] MIRunControlTargetAvailableTest.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/MIRunControlTargetAvailableTest.java?root=Tools_Project&revision=1.1&view=markup [*] Suite_6_8.java 1.5 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_6_8/Suite_6_8.java?root=Tools_Project&r1=1.4&r2=1.5 [+] MIRunControlTargetAvailableTest_6_8.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_6_8/MIRunControlTargetAvailableTest_6_8.java?root=Tools_Project&revision=1.1&view=markup [*] Suite_Remote_6_8.java 1.4 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_6_8/Suite_Remote_6_8.java?root=Tools_Project&r1=1.3&r2=1.4 [*] Suite_Remote_6_6.java 1.4 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_6_6/Suite_Remote_6_6.java?root=Tools_Project&r1=1.3&r2=1.4 [*] Suite_6_6.java 1.5 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_6_6/Suite_6_6.java?root=Tools_Project&r1=1.4&r2=1.5 [+] MIRunControlTargetAvailableTest_6_6.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_6_6/MIRunControlTargetAvailableTest_6_6.java?root=Tools_Project&revision=1.1&view=markup [+] TargetAvail.cc http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/data/launch/src/TargetAvail.cc?root=Tools_Project&revision=1.1&view=markup [+] MIRunControlTargetAvailableTest_7_2.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_2/MIRunControlTargetAvailableTest_7_2.java?root=Tools_Project&revision=1.1&view=markup [*] Suite_7_2.java 1.4 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_2/Suite_7_2.java?root=Tools_Project&r1=1.3&r2=1.4 [*] Suite_Remote_7_2.java 1.3 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_2/Suite_Remote_7_2.java?root=Tools_Project&r1=1.2&r2=1.3 [*] Suite_7_1.java 1.4 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_1/Suite_7_1.java?root=Tools_Project&r1=1.3&r2=1.4 [+] MIRunControlTargetAvailableTest_7_1.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_1/MIRunControlTargetAvailableTest_7_1.java?root=Tools_Project&revision=1.1&view=markup [*] Suite_Remote_7_1.java 1.3 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_1/Suite_Remote_7_1.java?root=Tools_Project&r1=1.2&r2=1.3 [*] Suite_Remote_6_7.java 1.4 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_6_7/Suite_Remote_6_7.java?root=Tools_Project&r1=1.3&r2=1.4 [*] Suite_6_7.java 1.5 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_6_7/Suite_6_7.java?root=Tools_Project&r1=1.4&r2=1.5 [+] MIRunControlTargetAvailableTest_6_7.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_6_7/MIRunControlTargetAvailableTest_6_7.java?root=Tools_Project&revision=1.1&view=markup [*] MIRunControl.java 1.33 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/mi/service/MIRunControl.java?root=Tools_Project&r1=1.32&r2=1.33 Created attachment 192038 [details]
Tests for non-stop mode
Here are some tests for non-stop mode for this problem. I just re-used the all-stop mode tests, but set non-stop to enabled.
We need more non-stop tests in the test suites in general, but this is a start.
Committed to HEAD.
*** cdt cvs genie on behalf of mkhouzam *** Bug 337893 - Tests for IMIRunControl.executeWithTargetAvailable, non-stop mode [+] MIRunControlNonStopTargetAvailableTest_7_2.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_2/MIRunControlNonStopTargetAvailableTest_7_2.java?root=Tools_Project&revision=1.1&view=markup [*] Suite_7_2.java 1.5 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_2/Suite_7_2.java?root=Tools_Project&r1=1.4&r2=1.5 [*] Suite_Remote_7_2.java 1.4 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_2/Suite_Remote_7_2.java?root=Tools_Project&r1=1.3&r2=1.4 [*] Suite_7_1.java 1.5 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_1/Suite_7_1.java?root=Tools_Project&r1=1.4&r2=1.5 [+] MIRunControlNonStopTargetAvailableTest_7_1.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_1/MIRunControlNonStopTargetAvailableTest_7_1.java?root=Tools_Project&revision=1.1&view=markup [*] Suite_Remote_7_1.java 1.4 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_1/Suite_Remote_7_1.java?root=Tools_Project&r1=1.3&r2=1.4 [*] Suite_Remote_7_0.java 1.6 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_0/Suite_Remote_7_0.java?root=Tools_Project&r1=1.5&r2=1.6 [*] Suite_7_0.java 1.6 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_0/Suite_7_0.java?root=Tools_Project&r1=1.5&r2=1.6 [+] MIRunControlNonStopTargetAvailableTest_7_0.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/tests_7_0/MIRunControlNonStopTargetAvailableTest_7_0.java?root=Tools_Project&revision=1.1&view=markup Created attachment 192202 [details]
Fix for non-stop mode, single process
This fix is for non-stop. It is pretty much the exact solution used for all-stop, except that it stops a single thread instead of the entire container, since we can do that in non-stop.
It does not yet handle multi-process, which I will do as another patch. The reason I'm doing it like this is to not break the API, as the next patch will show.
Committed this patch to HEAD.
*** cdt cvs genie on behalf of mkhouzam *** Bug 337893 - Support concurrent setting of bp on a running target for non-stop mode, single process [*] GDBRunControl_7_0_NS.java 1.29 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GDBRunControl_7_0_NS.java?root=Tools_Project&r1=1.28&r2=1.29 Created attachment 192418 [details]
Fix for multi-process non-stop
And here is the fix for Linux multi-process non-stop.
The logic is exactly the same except that I wrapped the entire handling of executeWithTargetAvailable() into another class, so that we can perform this operation for different processes at the same time.
I created a new GDBRunControl_7_2_NS service which allowed be not to break the API. Also, it made sense to have a new service version since the new logic was really for multi-process only.
So, this patch does add new APIs but everything is backward-compatible.
Committed to HEAD.
*** cdt cvs genie on behalf of mkhouzam *** Bug 337893: Support for setting breakpoints on a running target with multi-process [*] GDBRunControl_7_0_NS.java 1.32 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GDBRunControl_7_0_NS.java?root=Tools_Project&r1=1.31&r2=1.32 [*] GdbDebugServicesFactoryNS.java 1.4 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GdbDebugServicesFactoryNS.java?root=Tools_Project&r1=1.3&r2=1.4 [+] GDBRunControl_7_2_NS.java http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GDBRunControl_7_2_NS.java?root=Tools_Project&revision=1.1&view=markup There was a lot of complex changes in this bug. I don't feel comfortable singling out one committer for review because it may mean a lot of work. But if anyone has time to look at the changes, that would be appreciated. Created attachment 192419 [details]
Fix for JUnit tests
I forgot to include a small fix for the JUnit tests.
Committed to HEAD.
*** cdt cvs genie on behalf of mkhouzam *** Bug 337893: Setting breakpoints on a running target does not work with multi-process [*] MIRunControlTargetAvailableTest.java 1.2 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.tests.dsf.gdb/src/org/eclipse/cdt/tests/dsf/gdb/tests/MIRunControlTargetAvailableTest.java?root=Tools_Project&r1=1.1&r2=1.2 Created attachment 192606 [details]
Minor fix
I found a little bug in this solution for non-stop.
I use a map that stores the process for which we are doing an executeWithTargetAvailable() but I never remove the entry from the map once the operation is finished.
This patch fixes it.
Committed to HEAD.
*** cdt cvs genie on behalf of mkhouzam *** Bug 337893: Missing cleanup of map [*] GDBRunControl_7_2_NS.java 1.3 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GDBRunControl_7_2_NS.java?root=Tools_Project&r1=1.2&r2=1.3 While trying to fix Bug 340697, I realized that in on-stop mode, it is not sufficient to stop the process for which we want to do a breakpoint operation. We need to stop _all_ processes that share that same binary. So, when debugging two processes that used fork(), setting a breakpoint was failing because we were only stopping one of the two processes. The whole thing becomes simpler in what will be GDB 7.3 when using gdbserver, where it won't be necessary to interrupt at all. But for native GDB, this limitation has not been fixed yet. See http://sourceware.org/ml/gdb/2011-04/msg00032.html So, I need to change the fix to this bug for non-stop. I can either: 1- interrupt all processes that use the same binary as the one for which we want to set a breakpoint 2- interrupt all processes. I'm actually aiming for solution #2 because it is the one with the least impact. Although this may seem wasteful to interrupt all processes when not necessary, in truth it is not so much; when making a breakpoint operation in Eclipse, that operation is propagated to all processes anyway, so they will all need to be interrupted. The case where we are wasteful is when we start or stop debugging a process (starting a process, attaching to one, auto-attaching to one, detaching from one, terminating one); in those cases, we only want to apply the breakpoint operation to that one process and any other using the same binary. Personally, I think the wastefulness is not such a big deal for that case, but it can always be improved later on. I'm working on a patch now. Created attachment 192993 [details]
Fix to interrupt a thread for every process
This fix is actually simpler than the previous once since we can blindly interrupt a thread for every process. Therefore, I was able to put it directly in GDBRunControl_7_0_NS instead of having a special solution in GDBRunControl_7_2_NS.
There are API changes, but they are removing things that I had added in this very bug, so I'm going ahead with it.
The JUnit tests pass for this logic.
Committed to HEAD.
Fixed. Finally. *** cdt cvs genie on behalf of mkhouzam *** Bug 337893: Setting breakpoints on a running target does not work with multi-process. Instead of interrupt only the process of interest, we need to interrupt all processes that share the same binary as the process of interest. For simplicity, we interrupt all processes. [*] GDBRunControl_7_0_NS.java 1.34 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GDBRunControl_7_0_NS.java?root=Tools_Project&r1=1.33&r2=1.34 [*] GdbDebugServicesFactoryNS.java 1.5 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GdbDebugServicesFactoryNS.java?root=Tools_Project&r1=1.4&r2=1.5 [*] GDBRunControl_7_2_NS.java 1.4 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GDBRunControl_7_2_NS.java?root=Tools_Project&r1=1.3&r2=1.4 Created attachment 193212 [details]
GDB trace
I'm still seeing errors when I step through the code after attaching to a process in non-stop mode.
Warning:
Cannot insert breakpoint 0.
Error accessing memory address 0x69015b7: Input/output error.
In addition Step Over behaves as Step Into.
Attached a sanitized gdb trace.
(In reply to comment #24) > Created attachment 193212 [details] > GDB trace > > I'm still seeing errors when I step through the code after attaching to a > process in non-stop mode. > > Warning: > Cannot insert breakpoint 0. > Error accessing memory address 0x69015b7: Input/output error. > > In addition Step Over behaves as Step Into. > > Attached a sanitized gdb trace. Thanks Sergey. I am able to reproduce this problem. It seems to be a GDB bug. I have confirmed that the breakpoint is an internal one used by GDB when doing the step over and since it cannot be set, GDB stops on the first instructions. So, both issues are the same problem. I've posted the problem and the way to easily reproduce it on the GDB mailing list. This seems pretty serious at first glance. See: http://sourceware.org/ml/gdb/2011-04/msg00060.html Let's see what happens. (In reply to comment #25) > Thanks Sergey. I am able to reproduce this problem. It seems to be a GDB bug. > I have confirmed that the breakpoint is an internal one used by GDB when doing > the step over and since it cannot be set, GDB stops on the first instructions. > So, both issues are the same problem. > > I've posted the problem and the way to easily reproduce it on the GDB mailing > list. This seems pretty serious at first glance. > See: http://sourceware.org/ml/gdb/2011-04/msg00060.html > Let's see what happens. I posted a patch for GDB. Hopefully it will be approved and be part of the maintenance release GDB 7.2.1. http://sourceware.org/ml/gdb-patches/2011-04/msg00261.html (In reply to comment #26) > I posted a patch for GDB. Hopefully it will be approved and be part of the > maintenance release GDB 7.2.1. > > http://sourceware.org/ml/gdb-patches/2011-04/msg00261.html I got approval and committed the fix for GDB 7.2.1 and 7.3, which are both planned to be released soon. |